cqt-agent/expansion-packs/hubtel-workflow/utils/test-runner-validator.md

CQT Agent

# Test Runner & Validator

This utility automatically executes generated tests and validates their functionality, providing detailed feedback and remediation steps for any failures.

## Purpose
Execute all generated test suites (unit, integration, E2E) to ensure they work correctly before delivery, and provide actionable feedback for any issues encountered.

## Test Execution Pipeline

### Phase 1: Pre-Execution Validation

#### 1. Environment Setup Validation
```bash
#!/bin/bash
# Validate test environment setup

validate_environment() {
  local validation_report=()
  
  # Check Node.js and npm versions
  if command -v node &> /dev/null; then
    NODE_VERSION=$(node --version)
    echo "✓ Node.js version: $NODE_VERSION"
  else
    validation_report+=("❌ Node.js not installed. Install Node.js 18+ from https://nodejs.org")
  fi
  
  # Check test frameworks
  if npx playwright --version &> /dev/null; then
    PLAYWRIGHT_VERSION=$(npx playwright --version)
    echo "✓ Playwright version: $PLAYWRIGHT_VERSION"
  else
    validation_report+=("❌ Playwright not installed. Run: npm install -D @playwright/test")
  fi
  
  # Check Java for Karate tests
  if command -v java &> /dev/null; then
    JAVA_VERSION=$(java -version 2>&1 | head -n 1)
    echo "✓ Java version: $JAVA_VERSION"
  else
    validation_report+=("❌ Java not installed. Install Java 11+ for Karate tests")
  fi
  
  # Check database connectivity
  if nc -z localhost 5432 &> /dev/null; then
    echo "✓ Database connection available"
  else
    validation_report+=("⚠️ Database not accessible. Some integration tests may fail")
  fi
  
  return ${#validation_report[@]}
}
```

#### 2. Syntax Validation
```bash
validate_test_syntax() {
  local syntax_errors=()
  
  echo "Validating test file syntax..."
  
  # Validate TypeScript test files
  find tests -name "*.ts" -type f | while read -r file; do
    if ! npx tsc --noEmit "$file" 2>/dev/null; then
      syntax_errors+=("TypeScript syntax error in: $file")
    fi
  done
  
  # Validate Playwright tests
  if ! npx playwright test --dry-run --reporter=json 2>/dev/null; then
    syntax_errors+=("Playwright configuration or test syntax errors detected")
  fi
  
  # Validate Karate tests
  find tests -name "*.feature" -type f | while read -r file; do
    if ! java -jar karate.jar "$file" --dry-run 2>/dev/null; then
      syntax_errors+=("Karate feature syntax error in: $file")
    fi
  done
  
  if [ ${#syntax_errors[@]} -eq 0 ]; then
    echo "✓ All test files have valid syntax"
    return 0
  else
    echo "❌ Syntax validation failed:"
    printf '%s\n' "${syntax_errors[@]}"
    return 1
  fi
}
```

#### 3. Dependency Resolution
```bash
resolve_dependencies() {
  echo "Resolving test dependencies..."
  
  # Install npm dependencies
  if ! npm ci --silent; then
    echo "❌ Failed to install npm dependencies"
    echo "Action: Run 'npm install' and fix any dependency conflicts"
    return 1
  fi
  
  # Install Playwright browsers
  if ! npx playwright install --with-deps; then
    echo "❌ Failed to install Playwright browsers"
    echo "Action: Run 'npx playwright install' manually"
    return 1
  fi
  
  # Verify Karate JAR
  if [ ! -f "karate.jar" ]; then
    echo "⚠️ Karate JAR not found, downloading..."
    wget -O karate.jar https://github.com/karatelabs/karate/releases/download/v1.4.1/karate-1.4.1.jar
  fi
  
  echo "✓ All dependencies resolved successfully"
  return 0
}
```

### Phase 2: Test Execution

#### 1. Unit Test Execution
```bash
execute_unit_tests() {
  echo "Executing unit tests..."
  
  local unit_results_file="test-results/unit-test-results.json"
  mkdir -p test-results
  
  # Run unit tests with coverage
  if npm run test:unit -- --coverage --reporter=json --outputFile="$unit_results_file"; then
    local coverage=$(jq -r '.coverageMap.totals.lines.pct' "$unit_results_file" 2>/dev/null || echo "0")
    local passed_tests=$(jq -r '.numPassedTests' "$unit_results_file" 2>/dev/null || echo "0")
    local failed_tests=$(jq -r '.numFailedTests' "$unit_results_file" 2>/dev/null || echo "0")
    
    echo "✓ Unit tests completed: $passed_tests passed, $failed_tests failed"
    echo "✓ Code coverage: $coverage%"
    
    if (( $(echo "$coverage < 85" | bc -l) )); then
      echo "⚠️ Code coverage below 85% threshold"
    fi
    
    return $failed_tests
  else
    echo "❌ Unit tests failed to execute"
    return 1
  fi
}
```

#### 2. Integration Test Execution
```bash
execute_integration_tests() {
  echo "Executing integration tests..."
  
  local integration_results_file="test-results/integration-test-results.json"
  
  # Start test database if needed
  if docker ps | grep -q test-db; then
    echo "✓ Test database already running"
  else
    echo "Starting test database..."
    docker-compose -f docker-compose.test.yml up -d db
    sleep 10  # Wait for database to be ready
  fi
  
  # Run integration tests
  if java -jar karate.jar tests/integration --output "$integration_results_file"; then
    local total_scenarios=$(jq -r '.scenariosPassed + .scenariosFailed' "$integration_results_file" 2>/dev/null || echo "0")
    local passed_scenarios=$(jq -r '.scenariosPassed' "$integration_results_file" 2>/dev/null || echo "0")
    local failed_scenarios=$(jq -r '.scenariosFailed' "$integration_results_file" 2>/dev/null || echo "0")
    
    echo "✓ Integration tests completed: $passed_scenarios passed, $failed_scenarios failed"
    return $failed_scenarios
  else
    echo "❌ Integration tests failed to execute"
    return 1
  fi
}
```

#### 3. End-to-End Test Execution
```bash
execute_e2e_tests() {
  echo "Executing end-to-end tests..."
  
  local e2e_results_file="test-results/e2e-test-results.json"
  
  # Start application in test mode
  if ! pgrep -f "npm run start:test" > /dev/null; then
    echo "Starting application in test mode..."
    npm run start:test &
    APP_PID=$!
    sleep 15  # Wait for application to start
  fi
  
  # Run E2E tests
  if npx playwright test --reporter=json --output-file="$e2e_results_file"; then
    local total_tests=$(jq -r '.suites[].tests | length' "$e2e_results_file" | awk '{sum+=$1} END {print sum}' || echo "0")
    local passed_tests=$(jq -r '.suites[].tests[] | select(.outcome=="passed") | length' "$e2e_results_file" | wc -l || echo "0")
    local failed_tests=$(jq -r '.suites[].tests[] | select(.outcome=="failed") | length' "$e2e_results_file" | wc -l || echo "0")
    
    echo "✓ E2E tests completed: $passed_tests passed, $failed_tests failed"
    
    # Cleanup
    if [ -n "$APP_PID" ]; then
      kill $APP_PID 2>/dev/null
    fi
    
    return $failed_tests
  else
    echo "❌ E2E tests failed to execute"
    if [ -n "$APP_PID" ]; then
      kill $APP_PID 2>/dev/null
    fi
    return 1
  fi
}
```

### Phase 3: Results Analysis & Reporting

#### 1. Performance Validation
```bash
validate_performance() {
  echo "Validating performance metrics..."
  
  local performance_issues=()
  
  # Check E2E test execution times
  local avg_test_duration=$(jq -r '.suites[].tests[].duration' test-results/e2e-test-results.json | awk '{sum+=$1; count++} END {if(count>0) print sum/count; else print 0}')
  
  if (( $(echo "$avg_test_duration > 30000" | bc -l) )); then
    performance_issues+=("⚠️ Average E2E test duration (${avg_test_duration}ms) exceeds 30s threshold")
  fi
  
  # Check API response times
  local avg_api_response=$(jq -r '.scenarioResults[].stepResults[] | select(.step.text | contains("When method")) | .executionTime' test-results/integration-test-results.json | awk '{sum+=$1; count++} END {if(count>0) print sum/count; else print 0}')
  
  if (( $(echo "$avg_api_response > 2000" | bc -l) )); then
    performance_issues+=("⚠️ Average API response time (${avg_api_response}ms) exceeds 2s threshold")
  fi
  
  if [ ${#performance_issues[@]} -eq 0 ]; then
    echo "✓ All performance metrics within acceptable limits"
  else
    echo "Performance warnings:"
    printf '%s\n' "${performance_issues[@]}"
  fi
}
```

#### 2. Test Quality Analysis
```bash
analyze_test_quality() {
  echo "Analyzing test quality..."
  
  local quality_issues=()
  
  # Check for missing assertions
  local tests_without_assertions=$(grep -r "test\|scenario" tests/ | wc -l)
  local total_assertions=$(grep -r "expect\|assert\|match" tests/ | wc -l)
  local assertion_ratio=$(echo "scale=2; $total_assertions / $tests_without_assertions" | bc)
  
  if (( $(echo "$assertion_ratio < 3" | bc -l) )); then
    quality_issues+=("⚠️ Low assertion-to-test ratio ($assertion_ratio). Tests may lack sufficient validation")
  fi
  
  # Check for hardcoded values
  local hardcoded_values=$(grep -r "localhost\|127.0.0.1\|password123" tests/ | wc -l)
  if [ $hardcoded_values -gt 0 ]; then
    quality_issues+=("⚠️ Found $hardcoded_values hardcoded values in tests. Use configuration or test data files")
  fi
  
  # Check for test isolation
  local shared_state_tests=$(grep -r "beforeAll\|afterAll" tests/ | wc -l)
  local total_test_files=$(find tests/ -name "*.ts" -o -name "*.feature" | wc -l)
  
  if (( shared_state_tests > total_test_files * 2 )); then
    quality_issues+=("⚠️ Many tests share state. Consider improving test isolation")
  fi
  
  if [ ${#quality_issues[@]} -eq 0 ]; then
    echo "✓ Test quality analysis passed"
  else
    echo "Test quality warnings:"
    printf '%s\n' "${quality_issues[@]}"
  fi
}
```

### Phase 4: Comprehensive Reporting

#### 1. Generate Consolidated Report
```bash
generate_test_report() {
  local report_file="test-results/consolidated-report.json"
  local html_report="test-results/test-report.html"
  
  # Consolidate all test results
  jq -n \
    --slurpfile unit test-results/unit-test-results.json \
    --slurpfile integration test-results/integration-test-results.json \
    --slurpfile e2e test-results/e2e-test-results.json \
    '{
      timestamp: now | strftime("%Y-%m-%d %H:%M:%S"),
      summary: {
        unit: {
          passed: ($unit[0].numPassedTests // 0),
          failed: ($unit[0].numFailedTests // 0),
          coverage: ($unit[0].coverageMap.totals.lines.pct // 0)
        },
        integration: {
          passed: ($integration[0].scenariosPassed // 0),
          failed: ($integration[0].scenariosFailed // 0)
        },
        e2e: {
          passed: ($e2e[0].suites[].tests[] | select(.outcome=="passed") | length),
          failed: ($e2e[0].suites[].tests[] | select(.outcome=="failed") | length)
        }
      },
      details: {
        unit: $unit[0],
        integration: $integration[0],
        e2e: $e2e[0]
      }
    }' > "$report_file"
  
  # Generate HTML report
  generate_html_report "$report_file" "$html_report"
  
  echo "✓ Consolidated test report generated: $html_report"
}

generate_html_report() {
  local json_report=$1
  local html_report=$2
  
  cat > "$html_report" << 'EOF'
<!DOCTYPE html>
<html>
<head>
    <title>Test Execution Report</title>
    <style>
        body { font-family: Arial, sans-serif; margin: 20px; }
        .summary { background: #f5f5f5; padding: 15px; border-radius: 5px; margin-bottom: 20px; }
        .pass { color: #28a745; }
        .fail { color: #dc3545; }
        .warn { color: #ffc107; }
        table { border-collapse: collapse; width: 100%; }
        th, td { border: 1px solid #ddd; padding: 8px; text-align: left; }
        th { background-color: #f2f2f2; }
    </style>
</head>
<body>
    <h1>Test Execution Report</h1>
    <div id="report-content"></div>
    <script>
        // Load and display JSON report data
        fetch('consolidated-report.json')
            .then(response => response.json())
            .then(data => displayReport(data));
        
        function displayReport(data) {
            // Implementation for displaying test results
            document.getElementById('report-content').innerHTML = generateReportHTML(data);
        }
    </script>
</body>
</html>
EOF
}
```

## Main Execution Function

```bash
main_test_execution() {
  echo "🧪 Starting comprehensive test execution pipeline..."
  echo "================================================"
  
  local total_failures=0
  local execution_log="test-results/execution.log"
  mkdir -p test-results
  
  # Phase 1: Pre-execution validation
  echo "Phase 1: Environment Validation"
  if ! validate_environment; then
    echo "❌ Environment validation failed. Please fix the issues above and retry."
    return 1
  fi
  
  if ! validate_test_syntax; then
    echo "❌ Syntax validation failed. Please fix syntax errors and retry."
    return 1
  fi
  
  if ! resolve_dependencies; then
    echo "❌ Dependency resolution failed. Please fix dependency issues and retry."
    return 1
  fi
  
  # Phase 2: Test execution
  echo -e "\nPhase 2: Test Execution"
  
  execute_unit_tests
  total_failures=$((total_failures + $?))
  
  execute_integration_tests  
  total_failures=$((total_failures + $?))
  
  execute_e2e_tests
  total_failures=$((total_failures + $?))
  
  # Phase 3: Analysis
  echo -e "\nPhase 3: Results Analysis"
  validate_performance
  analyze_test_quality
  
  # Phase 4: Reporting
  echo -e "\nPhase 4: Report Generation"
  generate_test_report
  
  # Final summary
  echo -e "\n🎯 Test Execution Summary"
  echo "========================"
  if [ $total_failures -eq 0 ]; then
    echo "✅ All tests passed successfully!"
    echo "📊 Detailed report available at: test-results/test-report.html"
  else
    echo "❌ $total_failures test suite(s) had failures"
    echo "📊 Detailed report available at: test-results/test-report.html"
    echo "🔍 Run the test-failure-analyzer for detailed remediation steps"
  fi
  
  return $total_failures
}
```

## Usage Integration

This utility integrates with the auto-e2e-generator task:

1. **Automatic Execution**: After generating tests, immediately validates them
2. **Failure Detection**: Identifies syntax errors, environment issues, logic problems
3. **Actionable Feedback**: Provides specific steps to resolve each type of issue
4. **Quality Assurance**: Ensures generated tests meet quality standards before delivery

The test runner provides comprehensive validation and clear remediation guidance for any issues encountered during test execution.