agentic-data-stack-community/docs/templates-guide.md

AI Agentic Data Stack Framework - Community Edition. Open source data engineering framework with 4 core agents, essential templates, and 3-dimensional quality validation.

# Templates Guide - ADSF Community Edition

This comprehensive guide covers all 20 essential templates included in the Community Edition, organized by category and use case.

## 📋 Template Overview

The Community Edition includes 20 carefully selected templates that provide maximum value for data engineering and analytics projects. These templates represent the most commonly used patterns from the full enterprise library of 88 templates.

### Template Categories
- **Data Pipeline** (5 templates)
- **Quality & Monitoring** (4 templates)
- **Analytics & Reporting** (4 templates)
- **Project Management** (3 templates)
- **Infrastructure** (2 templates)
- **Business** (2 templates)

## 🔧 Data Pipeline Templates

### 1. data-pipeline-tmpl.yaml
**Purpose**: Core pipeline structure for ETL/ELT processes
**Agent**: Data Engineer
**Use Case**: Building standardized data pipelines

**Key Features:**
- Source system configuration
- Transformation logic definition
- Target system mapping
- Error handling patterns
- Monitoring integration

**Example Usage:**
```bash
agentic-data interactive
@data-engineer
*create-doc data-pipeline-tmpl
# Interactive prompts will guide you through pipeline configuration
```

**Template Structure:**
```yaml
pipeline:
  name: "Customer Data Pipeline"
  source:
    type: "database"
    connection: "postgresql://..."
    tables: ["customers", "orders"]
  
  transformations:
    - name: "clean_customer_data"
      type: "sql"
      script: "clean_customers.sql"
    
  target:
    type: "warehouse"
    schema: "analytics"
    table: "customers_clean"
  
  schedule:
    frequency: "daily"
    time: "02:00"
  
  monitoring:
    alerts: ["failure", "delay", "quality"]
    recipients: ["data-team@company.com"]
```

### 2. etl-patterns.yaml
**Purpose**: Reusable ETL pattern library
**Agent**: Data Engineer
**Use Case**: Implementing common transformation patterns

**Included Patterns:**
- **Slowly Changing Dimensions (SCD)**: Types 1, 2, and 3
- **Data Cleansing**: Standardization and validation
- **Aggregation Patterns**: Time-based and dimensional
- **Lookup Patterns**: Reference data integration
- **Error Handling**: Retry and dead letter patterns

**Example Pattern - SCD Type 2:**
```yaml
patterns:
  scd_type2:
    description: "Slowly Changing Dimension Type 2"
    implementation:
      - detect_changes: "MERGE statement with change detection"
      - create_versions: "Insert new records with version tracking"
      - close_old_versions: "Update end dates on previous versions"
    
    columns:
      business_key: "customer_id"
      version_columns: ["effective_date", "end_date", "current_flag"]
      
    sql_template: |
      MERGE target AS t
      USING source AS s ON t.customer_id = s.customer_id AND t.current_flag = 'Y'
      WHEN MATCHED AND (t.name != s.name OR t.email != s.email) THEN
        UPDATE SET end_date = CURRENT_DATE, current_flag = 'N'
      WHEN NOT MATCHED THEN
        INSERT (customer_id, name, email, effective_date, current_flag)
        VALUES (s.customer_id, s.name, s.email, CURRENT_DATE, 'Y')
```

### 3. data-ingestion-workflow.yaml
**Purpose**: Data ingestion workflow patterns
**Agent**: Data Engineer
**Use Case**: Standardizing data ingestion processes

**Workflow Types:**
- **Batch Ingestion**: Scheduled bulk data loads
- **Incremental Ingestion**: Change data capture patterns
- **API Ingestion**: REST API data consumption
- **File Ingestion**: CSV, JSON, Parquet file processing
- **Stream Ingestion**: Real-time data streaming (basic)

### 4. infrastructure-tmpl.yaml
**Purpose**: Infrastructure setup and configuration
**Agent**: Data Engineer
**Use Case**: Standardizing infrastructure deployment

**Components:**
- Database configuration
- Storage setup
- Network configuration
- Security settings
- Monitoring setup

### 5. deployment-tmpl.yaml
**Purpose**: Deployment process standardization
**Agent**: Data Engineer
**Use Case**: Consistent deployment procedures

**Deployment Stages:**
- Pre-deployment validation
- Deployment execution
- Post-deployment testing
- Rollback procedures
- Documentation updates

## 🔍 Quality & Monitoring Templates

### 6. quality-checks-tmpl.yaml
**Purpose**: 3-dimensional quality validation framework
**Agent**: Data Quality Engineer
**Use Case**: Implementing comprehensive quality checks

**Quality Dimensions:**
```yaml
quality_checks:
  completeness:
    - check_name: "record_count_validation"
      description: "Validate expected record counts"
      sql: |
        SELECT 
          COUNT(*) as actual_count,
          {expected_count} as expected_count,
          CASE WHEN COUNT(*) >= {threshold} * {expected_count} 
               THEN 'PASS' ELSE 'FAIL' END as status
        FROM {table_name}
    
    - check_name: "null_value_validation"
      description: "Check for unexpected null values"
      sql: |
        SELECT 
          column_name,
          COUNT(*) as null_count,
          COUNT(*) * 100.0 / (SELECT COUNT(*) FROM {table_name}) as null_percentage
        FROM {table_name}
        WHERE {column_name} IS NULL
  
  accuracy:
    - check_name: "data_type_validation"
      description: "Validate data types and formats"
      rules:
        - column: "email"
          pattern: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
        - column: "phone"
          pattern: "^\\+?[1-9]\\d{1,14}$"
    
    - check_name: "range_validation"
      description: "Validate numeric ranges"
      rules:
        - column: "age"
          min_value: 0
          max_value: 150
        - column: "order_amount"
          min_value: 0
  
  consistency:
    - check_name: "referential_integrity"
      description: "Validate foreign key relationships"
      sql: |
        SELECT COUNT(*) as orphaned_records
        FROM orders o
        LEFT JOIN customers c ON o.customer_id = c.customer_id
        WHERE c.customer_id IS NULL
```

### 7. data-profiling-tmpl.yaml
**Purpose**: Data exploration and understanding
**Agent**: Data Quality Engineer
**Use Case**: Initial data assessment and ongoing monitoring

**Profiling Categories:**
- **Basic Statistics**: Count, min, max, average
- **Distribution Analysis**: Histograms, percentiles
- **Pattern Analysis**: Common patterns, outliers
- **Relationship Analysis**: Correlations, dependencies

### 8. quality-monitoring-tmpl.yaml
**Purpose**: Ongoing quality monitoring setup
**Agent**: Data Quality Engineer
**Use Case**: Continuous quality validation

**Monitoring Components:**
- Real-time quality dashboards
- Alert configuration
- Quality trend analysis
- Exception reporting

### 9. monitoring-tmpl.yaml
**Purpose**: System and pipeline monitoring
**Agent**: Data Engineer
**Use Case**: Operational monitoring setup

**Monitoring Areas:**
- Pipeline execution monitoring
- System performance metrics
- Error tracking and alerting
- Business metrics tracking

## 📊 Analytics & Reporting Templates

### 10. data-analysis-tmpl.yaml
**Purpose**: Structured approach to data analysis
**Agent**: Data Analyst
**Use Case**: Standardizing analytical workflows

**Analysis Framework:**
```yaml
analysis_workflow:
  1_data_understanding:
    - data_source_review: "Understand data sources and quality"
    - exploratory_analysis: "Initial data exploration"
    - hypothesis_generation: "Develop analytical hypotheses"
  
  2_data_preparation:
    - data_cleaning: "Clean and standardize data"
    - feature_engineering: "Create analytical features"
    - data_transformation: "Apply necessary transformations"
  
  3_analysis_execution:
    - descriptive_analysis: "Summarize current state"
    - diagnostic_analysis: "Understand why something happened"
    - predictive_analysis: "Forecast future trends"
  
  4_insight_generation:
    - pattern_identification: "Identify key patterns"
    - insight_validation: "Validate insights with stakeholders"
    - actionable_recommendations: "Develop action items"
```

### 11. dashboard-tmpl.yaml
**Purpose**: Dashboard design and implementation
**Agent**: Data Analyst
**Use Case**: Creating effective business dashboards

**Dashboard Components:**
- Executive summary views
- Operational metrics
- Trend analysis charts
- Interactive filters
- Drill-down capabilities

### 12. data-visualization-tmpl.yaml
**Purpose**: Effective data visualization patterns
**Agent**: Data Analyst
**Use Case**: Creating compelling visualizations

**Visualization Types:**
- **Time Series**: Trend analysis and forecasting
- **Categorical**: Bar charts, pie charts, heatmaps
- **Comparative**: Side-by-side comparisons
- **Geographic**: Maps and location-based analysis
- **Network**: Relationship and flow diagrams

### 13. insight-report-tmpl.yaml
**Purpose**: Structured insight documentation
**Agent**: Data Analyst
**Use Case**: Communicating analytical findings

**Report Structure:**
```markdown
# Analysis Report: {Analysis Title}

## Executive Summary
- Key findings and recommendations
- Business impact and value

## Methodology
- Data sources and quality
- Analytical approach
- Limitations and assumptions

## Findings
- Detailed analysis results
- Supporting visualizations
- Statistical significance

## Recommendations
- Actionable next steps
- Implementation guidance
- Success metrics

## Appendix
- Technical details
- Additional charts
- Data dictionary
```

## 🎯 Project Management Templates

### 14. business-requirements-tmpl.yaml
**Purpose**: Comprehensive requirements documentation
**Agent**: Data Product Manager
**Use Case**: Gathering and documenting project requirements

**Requirements Framework:**
```yaml
business_requirements:
  project_overview:
    objective: "Primary business objective"
    scope: "Project scope and boundaries"
    success_criteria: "Definition of success"
    
  stakeholders:
    primary: 
      - name: "Business Owner"
        role: "Decision maker and sponsor"
        expectations: "Clear ROI and business value"
    
    secondary:
      - name: "End Users"
        role: "Daily system users"
        expectations: "Improved efficiency and insights"
  
  functional_requirements:
    - requirement_id: "FR001"
      description: "System must process customer data daily"
      priority: "High"
      acceptance_criteria:
        - "Process completes within 2 hours"
        - "99.9% data accuracy maintained"
        - "Zero data loss tolerance"
  
  non_functional_requirements:
    performance:
      - "System response time < 2 seconds"
      - "Support 100 concurrent users"
    
    security:
      - "Role-based access control"
      - "Data encryption at rest and in transit"
    
    availability:
      - "99.9% uptime SLA"
      - "Disaster recovery within 4 hours"
```

### 15. project-plan-tmpl.yaml
**Purpose**: Project planning and timeline management
**Agent**: Data Product Manager
**Use Case**: Structured project execution

**Project Phases:**
1. **Discovery**: Requirements and feasibility
2. **Design**: Architecture and technical design
3. **Development**: Implementation and testing
4. **Deployment**: Go-live and validation
5. **Operations**: Ongoing support and optimization

### 16. stakeholder-engagement-tmpl.yaml
**Purpose**: Stakeholder communication planning
**Agent**: Data Product Manager
**Use Case**: Managing stakeholder relationships

**Engagement Framework:**
- Stakeholder mapping and analysis
- Communication plans and schedules
- Meeting templates and agendas
- Status reporting formats
- Change management processes

## 🏗️ Infrastructure Templates

### 17. configuration-tmpl.yaml
**Purpose**: System configuration management
**Agent**: Data Engineer
**Use Case**: Standardizing configuration across environments

**Configuration Areas:**
- Database connections
- API endpoints
- Security settings
- Performance parameters
- Environment variables

### 18. documentation-tmpl.yaml
**Purpose**: Technical documentation standards
**Agent**: All Agents
**Use Case**: Maintaining comprehensive documentation

**Documentation Types:**
- Architecture documentation
- API documentation
- User guides
- Operations manuals
- Troubleshooting guides

## 💼 Business Templates

### 19. metric-definition-tmpl.yaml
**Purpose**: Business metrics standardization
**Agent**: Data Analyst
**Use Case**: Defining and tracking KPIs

**Metric Framework:**
```yaml
metrics:
  customer_lifetime_value:
    definition: "Total revenue expected from customer relationship"
    calculation: "Average Order Value × Purchase Frequency × Customer Lifespan"
    data_sources: ["orders", "customers"]
    update_frequency: "daily"
    business_owner: "Customer Success Team"
    
  monthly_active_users:
    definition: "Unique users who performed key action in last 30 days"
    calculation: "COUNT(DISTINCT user_id) WHERE last_activity >= CURRENT_DATE - 30"
    data_sources: ["user_activity"]
    update_frequency: "daily"
    business_owner: "Product Team"
```

### 20. value-mapping-tmpl.yaml
**Purpose**: Business value and ROI tracking
**Agent**: Data Product Manager
**Use Case**: Demonstrating project value

**Value Categories:**
- **Cost Savings**: Reduced operational costs
- **Revenue Growth**: Increased sales and efficiency
- **Risk Reduction**: Improved compliance and quality
- **Strategic Value**: Enhanced capabilities and insights

## 🚀 Interactive Template Usage Best Practices

### Using Templates with Agents
```bash
# List all available templates
agentic-data templates list

# View specific template details
agentic-data templates show business-requirements-tmpl

# Use agent for interactive template creation
agentic-data interactive
@data-product-manager
*create-doc business-requirements-tmpl
*exit
exit
```

### Selection Guidelines
1. **Start with Requirements**: Activate Morgan (Data Product Manager) for `business-requirements-tmpl`
2. **Plan Infrastructure**: Use Emma (Data Engineer) for infrastructure templates
3. **Implement Quality**: Use Quinn (Data Quality Engineer) for quality templates
4. **Build Analysis**: Use Riley (Data Analyst) for analysis and dashboard templates
5. **Interactive Creation**: Let agents guide template completion with expert prompts

### Customization Tips
1. **Adapt to Context**: Modify templates for your specific use case
2. **Maintain Standards**: Keep core structure while customizing content
3. **Version Control**: Track template modifications
4. **Share Learnings**: Contribute improvements back to community
5. **Document Changes**: Maintain change logs for customizations

### Integration Patterns
1. **Template Chaining**: Link related templates together
2. **Shared Components**: Reuse common elements across templates
3. **Consistent Naming**: Use consistent naming conventions
4. **Cross-References**: Link templates to related documentation
5. **Version Alignment**: Keep template versions synchronized

## 📚 Additional Resources

### Learning Path
1. **Getting Started**: Begin with getting-started.md
2. **Core Concepts**: Understand framework principles
3. **Template Deep-Dive**: Master individual templates
4. **Example Implementation**: Study the e-commerce example
5. **Advanced Usage**: Explore template customization

### Community Resources
- **GitHub Repository**: Latest templates and updates
- **Documentation Wiki**: Community-maintained guides
- **Discussion Forums**: Template usage questions
- **Example Gallery**: Real-world implementations
- **Contribution Guidelines**: How to contribute new templates

### Enterprise Upgrade
For access to 68 additional templates including industry-specific, ML-enhanced, and enterprise compliance templates:
- **Contact**: enterprise@agenticdatastack.com
- **Website**: https://www.agenticdatastack.com
- **Migration Guide**: Seamless upgrade path available

Start with these 20 essential templates to build powerful, standardized data solutions! 🎯