@cloudkinetix/bmad-enhanced/expansion-packs/ck-gitlab-cicd-automation/tasks/analyze-pipeline-failures.md

Cloud-Kinetix enhanced fork of BMAD-METHOD - Breakthrough Method of Agile AI-driven Development with robust versioning and unified validation.

# analyze-pipeline-failures

## Task: Intelligent GitLab Pipeline Failure Analysis

**Purpose**: Comprehensive analysis of pipeline failures with root cause identification, pattern detection, and actionable remediation strategies.

**When to Use**:

- After pipeline failures to identify root causes
- Recurring failure pattern analysis
- Pre-merge failure risk assessment
- DevOps process improvement initiatives

---

## Task Configuration

### Input Parameters

- `branch` (optional): Target branch to analyze (default: current branch)
- `analysis_depth` (optional): shallow, standard, deep (default: standard)
- `historical_analysis` (optional): Include pattern analysis across recent pipelines (default: true)
- `generate_report` (optional): Generate detailed failure report (default: true)
- `integration_alerts` (optional): Send failure alerts to integrated systems (default: true)

### Expected Outputs

- Detailed failure analysis with root cause identification
- Pattern detection across failed jobs
- Actionable remediation recommendations
- Integration alerts (JIRA bug reports, team notifications)
- Historical failure trend analysis

---

## Task Execution

### Phase 1: Failure Discovery and Context Gathering

```bash
# Verify authentication and branch context
echo "🔍 Starting Pipeline Failure Analysis..."
echo "======================================="

glab auth status || {
  echo "❌ GitLab CLI not authenticated"
  exit 1
}

ANALYSIS_BRANCH=${branch:-$(git branch --show-current)}
echo "📍 Analyzing branch: $ANALYSIS_BRANCH"

# Get current pipeline data
PIPELINE_DATA=$(glab ci get --output json --branch "$ANALYSIS_BRANCH" 2>/dev/null)

if [ $? -ne 0 ] || [ "$PIPELINE_DATA" = "" ]; then
  echo "❌ No pipeline data available for analysis"
  echo "   Branch: $ANALYSIS_BRANCH"
  exit 1
fi

# Check if pipeline actually failed
PIPELINE_STATUS=$(echo "$PIPELINE_DATA" | jq -r '.status')
if [ "$PIPELINE_STATUS" != "failed" ]; then
  echo "ℹ️ Pipeline status: $PIPELINE_STATUS"
  echo "   This analysis is optimized for failed pipelines"
  echo "   Continuing with general analysis..."
fi

echo "🎯 Pipeline ID: $(echo "$PIPELINE_DATA" | jq -r '.id')"
echo "📅 Created: $(echo "$PIPELINE_DATA" | jq -r '.created_at')"
echo "⏱️ Duration: $(echo "$PIPELINE_DATA" | jq -r '.duration // 0')s"
```

### Phase 2: Failed Job Identification and Categorization

```bash
echo ""
echo "🚨 Failed Job Analysis:"
echo "======================"

# Extract failed jobs
FAILED_JOBS=$(echo "$PIPELINE_DATA" | jq -r '.jobs[] | select(.status == "failed") | .name')

if [ -z "$FAILED_JOBS" ]; then
  echo "✅ No failed jobs detected in current pipeline"

  # Check for other concerning statuses
  CONCERNING_JOBS=$(echo "$PIPELINE_DATA" | jq -r '.jobs[] | select(.status == "canceled" or .status == "skipped") | "\(.name): \(.status)"')
  if [ -n "$CONCERNING_JOBS" ]; then
    echo ""
    echo "⚠️ Jobs with concerning statuses:"
    echo "$CONCERNING_JOBS"
  fi
else
  echo "❌ Failed jobs detected: $(echo "$FAILED_JOBS" | wc -l)"
  echo ""

  # Categorize failed jobs by stage and type
  echo "📊 Failure Categorization:"
  echo "--------------------------"

  echo "$FAILED_JOBS" | while read job_name; do
    if [ -n "$job_name" ]; then
      # Get job details
      JOB_DATA=$(echo "$PIPELINE_DATA" | jq -r ".jobs[] | select(.name == \"$job_name\")")
      JOB_STAGE=$(echo "$JOB_DATA" | jq -r '.stage // "unknown"')
      JOB_ID=$(echo "$JOB_DATA" | jq -r '.id')
      JOB_DURATION=$(echo "$JOB_DATA" | jq -r '.duration // 0')

      echo ""
      echo "🔴 Job: $job_name"
      echo "   Stage: $JOB_STAGE"
      echo "   Duration: ${JOB_DURATION}s"
      echo "   Job ID: $JOB_ID"

      # Categorize failure type based on job name patterns
      case "$job_name" in
        *test*|*spec*|*check*)
          echo "   Category: 🧪 Test Failure"
          ;;
        *build*|*compile*)
          echo "   Category: 🔨 Build Failure"
          ;;
        *deploy*|*release*)
          echo "   Category: 🚀 Deployment Failure"
          ;;
        *lint*|*format*|*style*)
          echo "   Category: 📏 Code Quality Failure"
          ;;
        *security*|*scan*)
          echo "   Category: 🔒 Security Scan Failure"
          ;;
        *)
          echo "   Category: ❓ General Failure"
          ;;
      esac
    fi
  done
fi
```

### Phase 3: Root Cause Analysis Through Log Analysis

```bash
echo ""
echo "🔬 Root Cause Analysis:"
echo "======================"

if [ -n "$FAILED_JOBS" ]; then
  echo "$FAILED_JOBS" | while read job_name; do
    if [ -n "$job_name" ]; then
      echo ""
      echo "--- Analyzing: $job_name ---"

      # Get job ID for log retrieval
      JOB_ID=$(echo "$PIPELINE_DATA" | jq -r ".jobs[] | select(.name == \"$job_name\") | .id")

      if [ "$JOB_ID" != "null" ]; then
        # Retrieve and analyze logs
        echo "📝 Retrieving job logs..."
        JOB_LOGS=$(glab ci trace "$JOB_ID" 2>/dev/null)

        if [ -n "$JOB_LOGS" ]; then
          # Pattern-based failure analysis
          echo "🔍 Pattern Analysis:"

          # Check for common failure patterns
          if echo "$JOB_LOGS" | grep -qi "permission denied\|access denied"; then
            echo "   🔒 ROOT CAUSE: Permission/Access Issue"
            echo "   💡 RECOMMENDATION: Check file permissions, credentials, or access rights"

          elif echo "$JOB_LOGS" | grep -qi "timeout\|timed out"; then
            echo "   ⏱️ ROOT CAUSE: Timeout Issue"
            echo "   💡 RECOMMENDATION: Increase timeout values or optimize job performance"

          elif echo "$JOB_LOGS" | grep -qi "dependency.*not found\|module.*not found\|package.*not found"; then
            echo "   📦 ROOT CAUSE: Missing Dependency"
            echo "   💡 RECOMMENDATION: Update dependencies, check package.json/requirements.txt"

          elif echo "$JOB_LOGS" | grep -qi "test.*failed\|assertion.*failed\|expected.*but got"; then
            echo "   🧪 ROOT CAUSE: Test Assertion Failure"
            echo "   💡 RECOMMENDATION: Review failing test cases and fix application logic"

          elif echo "$JOB_LOGS" | grep -qi "compilation.*error\|build.*failed\|syntax.*error"; then
            echo "   🔨 ROOT CAUSE: Compilation/Build Error"
            echo "   💡 RECOMMENDATION: Fix syntax errors or build configuration issues"

          elif echo "$JOB_LOGS" | grep -qi "out of memory\|memory.*exceeded"; then
            echo "   💾 ROOT CAUSE: Memory Limitation"
            echo "   💡 RECOMMENDATION: Increase memory allocation or optimize memory usage"

          elif echo "$JOB_LOGS" | grep -qi "network.*error\|connection.*failed\|host.*unreachable"; then
            echo "   🌐 ROOT CAUSE: Network Connectivity Issue"
            echo "   💡 RECOMMENDATION: Check network configuration and external service availability"

          elif echo "$JOB_LOGS" | grep -qi "lint.*error\|format.*error\|style.*violation"; then
            echo "   📏 ROOT CAUSE: Code Quality/Style Issue"
            echo "   💡 RECOMMENDATION: Run linter locally and fix code style violations"

          else
            echo "   ❓ ROOT CAUSE: Unclassified Failure"
            echo "   💡 RECOMMENDATION: Manual log review required"
          fi

          # Show relevant log excerpt
          echo ""
          echo "📄 Relevant Log Excerpt (last 10 lines):"
          echo "$JOB_LOGS" | tail -10 | sed 's/^/   /'

          # Advanced analysis for deep mode
          if [ "$analysis_depth" = "deep" ]; then
            echo ""
            echo "🔬 Deep Analysis:"

            # Error frequency analysis
            ERROR_COUNT=$(echo "$JOB_LOGS" | grep -ci "error\|failed\|exception")
            WARNING_COUNT=$(echo "$JOB_LOGS" | grep -ci "warning\|warn")

            echo "   Error mentions: $ERROR_COUNT"
            echo "   Warning mentions: $WARNING_COUNT"

            # Extract specific error messages
            echo "   Key error messages:"
            echo "$JOB_LOGS" | grep -i "error\|failed\|exception" | tail -5 | sed 's/^/     /'
          fi

        else
          echo "   ⚠️ No logs available for analysis"
        fi

        echo ""
        echo "🔗 Full logs command: glab ci trace $JOB_ID"
      fi
    fi
  done
fi
```

### Phase 4: Historical Pattern Analysis

```bash
if [ "$historical_analysis" = "true" ]; then
  echo ""
  echo "📈 Historical Pattern Analysis:"
  echo "==============================="

  # Use pipeline analyzer for pattern detection
  source .bmad-core/utils/pipeline-analyzer.md

  echo "🔍 Analyzing failure patterns across recent pipelines..."
  analyze_failure_patterns "$ANALYSIS_BRANCH"

  # Additional historical context
  echo ""
  echo "📊 Recent Pipeline Health Trends:"
  calculate_pipeline_health "$ANALYSIS_BRANCH"
fi
```

### Phase 5: Environment and Configuration Analysis

```bash
echo ""
echo "⚙️ Environment & Configuration Analysis:"
echo "========================================"

# Check CI configuration
if [ -f ".gitlab-ci.yml" ]; then
  echo "📋 GitLab CI Configuration Status:"

  # Validate CI configuration
  CONFIG_VALIDATION=$(glab ci lint 2>/dev/null)
  if [ $? -eq 0 ]; then
    echo "   ✅ .gitlab-ci.yml syntax is valid"
  else
    echo "   ❌ .gitlab-ci.yml has syntax errors"
    echo "   💡 RECOMMENDATION: Fix CI configuration syntax"
  fi

  # Analyze CI file for common issues
  echo ""
  echo "🔍 Configuration Analysis:"

  # Check for resource limitations
  if grep -q "memory\|cpu\|resources" .gitlab-ci.yml; then
    echo "   ℹ️ Resource constraints configured"
  else
    echo "   ⚠️ No explicit resource constraints found"
    echo "   💡 RECOMMENDATION: Consider adding resource limits to prevent resource-related failures"
  fi

  # Check for timeout configurations
  if grep -q "timeout" .gitlab-ci.yml; then
    echo "   ℹ️ Custom timeouts configured"
  else
    echo "   ⚠️ Using default timeouts"
    echo "   💡 RECOMMENDATION: Consider explicit timeout configuration for long-running jobs"
  fi

else
  echo "❌ No .gitlab-ci.yml found"
  echo "   💡 RECOMMENDATION: Ensure CI configuration file exists and is properly named"
fi

# Check repository context
echo ""
echo "📁 Repository Context:"
echo "   Branch: $ANALYSIS_BRANCH"
echo "   Recent commits:"
git log --oneline -5 | sed 's/^/     /'

# Check for environment variables or secrets issues
echo ""
echo "🔐 Environment Analysis:"
echo "   💡 Common environment-related failure causes:"
echo "     - Missing required environment variables"
echo "     - Expired or invalid secrets/tokens"
echo "     - Incorrect environment-specific configurations"
echo "   🔍 Review pipeline variables in GitLab project settings"
```

### Phase 6: Integration Alerts and JIRA Updates

```bash
if [ "$integration_alerts" = "true" ]; then
  echo ""
  echo "🔗 Integration Alerts:"
  echo "===================="

  # Use integration bridge for cross-pack coordination
  source .bmad-core/utils/gitlab-integration-bridge.md

  # Detect integration opportunities
  detect_expansion_packs
  auto_detect_integration_context

  # JIRA integration for failure reporting
  if [ "$JIRA_INTEGRATION" = "true" ] && [ -n "$DETECTED_JIRA_ISSUES" ]; then
    echo ""
    echo "🎯 JIRA Integration Alert:"
    echo "   Detected JIRA issues: $DETECTED_JIRA_ISSUES"
    echo "   💡 RECOMMENDATION: Update JIRA issues with failure analysis"
    echo "   🔄 Creating JIRA-compatible failure report..."

    # Prepare JIRA failure report
    JIRA_FAILURE_REPORT="h3. ❌ CI Pipeline Failure Analysis

*Branch:* $ANALYSIS_BRANCH
*Pipeline ID:* $(echo "$PIPELINE_DATA" | jq -r '.id')
*Failure Time:* $(date)

h4. Failed Jobs:
$(echo "$FAILED_JOBS" | while read job; do echo "* $job"; done)

h4. Root Cause Analysis:
See pipeline failure analysis for detailed investigation.

*Pipeline URL:* [View Pipeline|$(echo "$PIPELINE_DATA" | jq -r '.web_url')]
"

    echo "   📝 JIRA update prepared - use jira agent to apply"
  fi

  # Parallel development coordination
  if [ "$PARALLEL_DEV_INTEGRATION" = "true" ]; then
    echo ""
    echo "🔀 Parallel Development Alert:"
    echo "   💡 This failure may block parallel development merge"
    echo "   🔄 Coordinating with other worktrees..."
    coordinate_parallel_ci
  fi
fi
```

### Phase 7: Failure Report Generation

```bash
if [ "$generate_report" = "true" ]; then
  echo ""
  echo "📄 Generating Comprehensive Failure Report:"
  echo "==========================================="

  REPORT_FILE="pipeline_failure_analysis_$(date +%Y%m%d_%H%M%S).md"

  # Generate detailed markdown report
  cat > "$REPORT_FILE" << EOF
# Pipeline Failure Analysis Report

**Generated:** $(date)
**Branch:** $ANALYSIS_BRANCH
**Pipeline ID:** $(echo "$PIPELINE_DATA" | jq -r '.id')
**Analysis Depth:** $analysis_depth

## Executive Summary

$(if [ -n "$FAILED_JOBS" ]; then
  echo "❌ **Status:** Pipeline failed with $(echo "$FAILED_JOBS" | wc -l) failed job(s)"
  echo ""
  echo "**Failed Jobs:**"
  echo "$FAILED_JOBS" | while read job; do echo "- $job"; done
else
  echo "ℹ️ **Status:** Pipeline analysis completed (status: $PIPELINE_STATUS)"
fi)

## Detailed Analysis

### Pipeline Overview
- **ID:** $(echo "$PIPELINE_DATA" | jq -r '.id')
- **Status:** $(echo "$PIPELINE_DATA" | jq -r '.status')
- **Duration:** $(echo "$PIPELINE_DATA" | jq -r '.duration // 0')s
- **Created:** $(echo "$PIPELINE_DATA" | jq -r '.created_at')
- **URL:** $(echo "$PIPELINE_DATA" | jq -r '.web_url')

### Job Analysis
$(echo "$PIPELINE_DATA" | jq -r '.jobs[] | "- **\(.name):** \(.status) (\(.stage)) - \(.duration // 0)s"')

## Recommendations

$(if [ -n "$FAILED_JOBS" ]; then
  echo "### Immediate Actions"
  echo "1. 🔍 Review failed job logs using: \`glab ci trace <job-id>\`"
  echo "2. 🔧 Address root causes identified in analysis"
  echo "3. 📝 Update code and push fixes"
  echo "4. 🔄 Monitor new pipeline execution"
  echo ""
  echo "### Integration Actions"
  if [ "$JIRA_INTEGRATION" = "true" ]; then
    echo "- 🎯 Update JIRA issues with failure details"
  fi
  if [ "$PARALLEL_DEV_INTEGRATION" = "true" ]; then
    echo "- 🔀 Coordinate with parallel development team"
  fi
else
  echo "### General Recommendations"
  echo "- ✅ Pipeline appears healthy"
  echo "- 🔄 Continue with normal development workflow"
fi)

## Technical Details

### Environment Context
- **Repository:** $(git remote get-url origin 2>/dev/null || echo "Local repository")
- **Branch:** $ANALYSIS_BRANCH
- **Recent Commits:**
$(git log --oneline -3 | sed 's/^/  /')

---
*Report generated by GitLab CI/CD Automation - analyze-pipeline-failures task*
EOF

  echo "📄 Report saved: $REPORT_FILE"
  echo "📊 Report contains comprehensive failure analysis and recommendations"
fi
```

### Phase 8: Next Steps and Recovery Guidance

```bash
echo ""
echo "🎯 Next Steps & Recovery Guidance:"
echo "================================="

if [ -n "$FAILED_JOBS" ]; then
  echo "🚨 IMMEDIATE ACTIONS REQUIRED:"
  echo ""
  echo "1. 🔍 INVESTIGATE - Review the root cause analysis above"
  echo "2. 🔧 FIX - Address identified issues in your code"
  echo "3. 📝 COMMIT - Push fixes to trigger new pipeline"
  echo "4. 🔄 MONITOR - Watch new pipeline execution"

  if [ "$JIRA_INTEGRATION" = "true" ] && [ -n "$DETECTED_JIRA_ISSUES" ]; then
    echo "5. 🎯 UPDATE JIRA - Inform stakeholders of resolution progress"
  fi

  echo ""
  echo "🔧 COMMON QUICK FIXES:"
  echo "- Permission issues: Check file permissions and access credentials"
  echo "- Dependency issues: Update package.json, requirements.txt, or similar"
  echo "- Test failures: Review and fix failing test cases"
  echo "- Build errors: Fix syntax errors and compilation issues"
  echo "- Timeout issues: Optimize job performance or increase timeout limits"

else
  echo "✅ NO IMMEDIATE ACTIONS REQUIRED"
  echo ""
  echo "📈 OPTIMIZATION OPPORTUNITIES:"
  echo "- Review pipeline performance for optimization"
  echo "- Consider adding more comprehensive tests"
  echo "- Evaluate CI/CD configuration for improvements"
fi

echo ""
echo "🆘 NEED HELP?"
echo "- Review full job logs: glab ci trace <job-id>"
echo "- Check GitLab project CI/CD settings"
echo "- Consult team DevOps guidelines"
echo "- Use glab agent for interactive assistance"

echo ""
echo "✅ ANALYSIS COMPLETE"
echo "📊 Use the generated insights to improve pipeline reliability"
```

---

## Integration Hooks

### JIRA Integration Points

- Automatic bug report creation for pipeline failures
- Failure analysis details added to JIRA comments
- Status updates for development progress tracking

### Parallel Development Integration Points

- Failure impact assessment across worktrees
- Merge blocking alerts for failed pipelines
- Coordination recommendations for team workflow

### Core BMAD Integration Points

- Failure analysis integration with development workflows
- Root cause insights for architecture decisions
- Quality gate integration for story completion criteria

---

## Success Criteria

- ✅ Successfully identifies and categorizes all pipeline failures
- ✅ Provides accurate root cause analysis with actionable recommendations
- ✅ Integrates failure alerts with available expansion packs
- ✅ Generates comprehensive failure reports for documentation
- ✅ Operates autonomously with intelligent pattern recognition
- ✅ Delivers clear next steps for failure resolution

## Dependencies

- **GitLab CLI** (`glab`) with authentication
- **Utilities**: ci-status-parser, pipeline-analyzer, gitlab-integration-bridge
- **Optional**: JIRA integration, parallel-dev integration
- **Git repository** with GitLab remote and CI configuration