@cloudkinetix/bmad-enhanced/expansion-packs/ck-gitlab-cicd-automation/utils/gitlab-commands.md

Cloud-Kinetix enhanced fork of BMAD-METHOD - Breakthrough Method of Agile AI-driven Development with robust versioning and unified validation.

# GitLab Commands Utility

## Non-Interactive GitLab CLI Commands for Autonomous CI/CD Operations

This utility contains verified GitLab CLI commands that work reliably in non-interactive mode for CI/CD monitoring and management. All commands have been tested and verified to work without user interaction prompts.

## ⚠️ Critical Update: Command Reliability & Fallback Strategies

Based on production usage feedback, many `glab` commands have reliability issues. This utility now includes:

- **Cascading command strategies** - Multiple fallback methods for each operation
- **Direct API integration** - Bypass glab when it fails
- **Permission validation** - Test access before attempting operations
- **Debug mode** - Detailed logging for troubleshooting

## Core Principles for Non-Interactive Operations

1. **Always use `--output json` or `--output table` to force structured output**
2. **Always use `2>/dev/null` to suppress interactive prompts and error dialogues**
3. **Always pipe JSON through `jq` for clean data extraction**
4. **Always use `|| echo "fallback message"` for error handling**
5. **Always specify `--branch` explicitly when needed**
6. **Always implement fallback strategies for critical operations**
7. **Always validate permissions before attempting detailed operations**

## Debug Mode

Enable debug mode for detailed command execution logging:

```bash
export GITLAB_DEBUG=true
```

### Debug Logging Function

```bash
# Debug logging helper
debug_log() {
  if [[ "$GITLAB_DEBUG" == "true" ]]; then
    echo "[DEBUG] $(date '+%Y-%m-%d %H:%M:%S') - $*" >&2
  fi
}

# Execute command with debug logging
execute_glab_command() {
  local cmd="$*"
  debug_log "Executing: $cmd"

  local output
  local exit_code

  output=$(eval "$cmd" 2>&1)
  exit_code=$?

  debug_log "Exit code: $exit_code"
  debug_log "Output: $output"

  if [[ $exit_code -ne 0 ]]; then
    debug_log "Command failed: $cmd"
    debug_log "Error: $output"
  fi

  echo "$output"
  return $exit_code
}
```

## Permission Validation

### Validate GitLab Access Permissions

```bash
validate_gitlab_permissions() {
  echo "🔍 Validating GitLab access permissions..."

  # Test basic authentication
  if glab auth status >/dev/null 2>&1; then
    echo "✅ Authentication: Valid"
  else
    echo "❌ Authentication: Failed"
    return 1
  fi

  # Test project access
  if glab ci list >/dev/null 2>&1; then
    echo "✅ Project Access: Valid"
  else
    echo "❌ Project Access: Failed"
    return 1
  fi

  # Test pipeline detail access (often fails)
  local test_pipeline=$(glab ci list 2>/dev/null | head -2 | tail -1 | awk '{print $4}' | sed 's/[()]//g')
  if [[ -n "$test_pipeline" ]]; then
    if glab ci view "$test_pipeline" >/dev/null 2>&1; then
      echo "✅ Pipeline Details: Accessible"
    else
      echo "⚠️  Pipeline Details: Limited (list only)"
    fi
  fi

  # Test job trace access
  echo "ℹ️  Job trace access will be tested when needed"
}
```

## Authentication & Project Info

### Check Authentication Status

```bash
# Verify GitLab CLI authentication (non-interactive)
glab auth status
```

### Get Project Information

```bash
# Get current project info
glab repo view --output json | jq -r '.path_with_namespace // "No repo info"'

# Get project statistics
glab repo view --output json | jq -r '.statistics // "No stats"'

# Check project permissions
glab repo view --output json | jq -r '.permissions // "No permissions info"'
```

## Pipeline Operations

### Latest Pipeline Information (With Fallback Strategies)

```bash
# Robust pipeline information retrieval with cascading fallbacks
get_pipeline_info_robust() {
  local branch="${1:-$(git branch --show-current)}"
  debug_log "Getting pipeline info for branch: $branch"

  # Method 1: Try glab ci get (often works)
  local pipeline_info=$(glab ci get --output json --branch "$branch" 2>/dev/null)
  if [[ -n "$pipeline_info" ]] && [[ "$pipeline_info" != "null" ]]; then
    debug_log "Method 1 successful: glab ci get"
    echo "$pipeline_info"
    return 0
  fi

  # Method 2: Try glab ci status + list parsing
  debug_log "Method 1 failed, trying method 2: glab ci list parsing"
  local pipeline_id=$(glab ci list 2>/dev/null | grep "$branch" | head -1 | awk '{print $4}' | sed 's/[()]//g')
  if [[ -n "$pipeline_id" ]]; then
    # Try to get more info using the API
    if [[ -n "$GITLAB_TOKEN" ]] && [[ -n "$CI_PROJECT_ID" ]]; then
      local api_info=$(curl -s -H "Authorization: Bearer $GITLAB_TOKEN" \
        "${GITLAB_HOST:-https://gitlab.com}/api/v4/projects/$CI_PROJECT_ID/pipelines/$pipeline_id" 2>/dev/null)
      if [[ -n "$api_info" ]]; then
        debug_log "Method 2 successful: API call with pipeline ID"
        echo "$api_info"
        return 0
      fi
    fi
    # Fallback to basic info
    echo "{\"id\": \"$pipeline_id\", \"status\": \"$(glab ci status --branch $branch 2>/dev/null || echo unknown)\", \"ref\": \"$branch\"}"
    return 0
  fi

  # Method 3: Basic fallback
  debug_log "All methods failed, returning basic status"
  echo "{\"error\": \"Limited GitLab access\", \"status\": \"unknown\", \"ref\": \"$branch\"}"
  return 1
}

# Get latest pipeline on current branch
get_pipeline_info_robust | jq -r '"\(.id // "N/A") \(.status // "unknown") \(.ref // "N/A") \(.duration // "N/A") \(.web_url // "N/A")"'

# Get latest pipeline on specific branch
get_pipeline_info_robust develop | jq -r '"\(.id // "N/A") \(.status // "unknown") \(.ref // "N/A") \(.duration // "N/A")"'

# Get pipeline ID only (for chaining commands)
get_pipeline_info_robust develop | jq -r '.id // "No pipeline"'
```

### Pipeline Status Monitoring

```bash
# Check pipeline status on specific branch
glab ci status --branch develop 2>/dev/null || echo "No pipeline status available"

# Get pipeline status in structured format
glab ci get --output json 2>/dev/null | jq -r '.status // "unknown"' 2>/dev/null

# Monitor running pipelines
glab ci get --output json 2>/dev/null | jq -r 'select(.status == "running") | "\(.id) \(.status) \(.ref)"' 2>/dev/null || echo "No running pipelines"
```

## Job Operations

### Job Information Extraction

```bash
# Get all jobs from latest pipeline
glab ci get --output json 2>/dev/null | jq -r '.jobs[] | "\(.id) \(.name) \(.status)"' 2>/dev/null || echo "No jobs found"

# Get specific job status
glab ci get --output json 2>/dev/null | jq -r '.jobs[] | select(.name == "test") | "\(.id) \(.status)"' 2>/dev/null || echo "Job not found"

# Get failed jobs only
glab ci get --output json 2>/dev/null | jq -r '.jobs[] | select(.status == "failed") | "\(.id) \(.name) \(.status)"' 2>/dev/null || echo "No failed jobs"

# Get job IDs for log viewing
glab ci get --output json 2>/dev/null | jq -r '.jobs[].id' 2>/dev/null | head -5
```

### Job Log Analysis (Critical Update)

⚠️ **Important Discovery**: `glab ci trace` only works with numeric job IDs, NOT job names!

```bash
# ❌ DOES NOT WORK - Common misconception
# glab ci trace "job-name"  # This will fail!
# glab ci trace pipeline_id:job_name  # This will fail!

# ✅ CORRECT APPROACH - Get job ID first, then trace
get_job_logs() {
  local pipeline_id="$1"
  local job_name="$2"

  debug_log "Getting logs for job '$job_name' in pipeline $pipeline_id"

  # Step 1: Get job ID using API (most reliable method)
  local job_id
  if [[ -n "$GITLAB_TOKEN" ]] && command -v glab >/dev/null 2>&1; then
    job_id=$(glab api "projects/:id/pipelines/$pipeline_id/jobs" 2>/dev/null | \
      jq -r ".[] | select(.name == \"$job_name\") | .id" 2>/dev/null)
  fi

  if [[ -z "$job_id" ]]; then
    # Fallback: Try to get from glab ci get
    job_id=$(glab ci get --output json 2>/dev/null | \
      jq -r ".jobs[] | select(.name == \"$job_name\") | .id" 2>/dev/null)
  fi

  if [[ -z "$job_id" ]]; then
    echo "❌ Job '$job_name' not found in pipeline $pipeline_id"
    return 1
  fi

  debug_log "Found job ID: $job_id for job name: $job_name"

  # Step 2: Get trace using numeric job ID
  local logs
  logs=$(glab ci trace "$job_id" 2>&1)
  local exit_code=$?

  if [[ $exit_code -eq 0 ]]; then
    echo "$logs"
    return 0
  else
    # Fallback to API if glab fails
    if [[ -n "$GITLAB_TOKEN" ]] && [[ -n "$CI_PROJECT_ID" ]]; then
      debug_log "glab trace failed, trying API fallback"
      curl -s -H "Authorization: Bearer $GITLAB_TOKEN" \
        "${GITLAB_HOST:-https://gitlab.com}/api/v4/projects/$CI_PROJECT_ID/jobs/$job_id/trace" 2>/dev/null
    else
      echo "❌ Failed to retrieve logs for job $job_id. Error: $logs"
      return 1
    fi
  fi
}

# Example usage:
# get_job_logs "1909685353" "services_and_e2e_tests"

# Get logs for all failed jobs
get_failed_job_logs() {
  local pipeline_id="${1:-$(get_pipeline_info_robust | jq -r '.id // ""')}"

  if [[ -z "$pipeline_id" ]]; then
    echo "❌ No pipeline ID available"
    return 1
  fi

  # Get failed job names and IDs
  local failed_jobs
  if command -v glab >/dev/null 2>&1 && [[ -n "$GITLAB_TOKEN" ]]; then
    failed_jobs=$(glab api "projects/:id/pipelines/$pipeline_id/jobs" 2>/dev/null | \
      jq -r '.[] | select(.status == "failed") | "\(.id):\(.name)"' 2>/dev/null)
  else
    failed_jobs=$(glab ci get --output json 2>/dev/null | \
      jq -r '.jobs[] | select(.status == "failed") | "\(.id):\(.name)"' 2>/dev/null)
  fi

  if [[ -z "$failed_jobs" ]]; then
    echo "✅ No failed jobs found"
    return 0
  fi

  # Get logs for each failed job
  while IFS=: read -r job_id job_name; do
    echo "=== Logs for failed job: $job_name (ID: $job_id) ==="
    glab ci trace "$job_id" 2>&1 | tail -100 || echo "Failed to get logs"
    echo ""
  done <<< "$failed_jobs"
}

# Quick helper to get last N lines of job log
get_job_tail() {
  local pipeline_id="$1"
  local job_name="$2"
  local lines="${3:-100}"

  get_job_logs "$pipeline_id" "$job_name" | tail -"$lines"
}
```

## Pipeline Management Operations

### Pipeline Control

```bash
# Cancel running pipeline
glab ci cancel --pipeline-id PIPELINE_ID 2>/dev/null || echo "Cannot cancel pipeline"

# Retry failed pipeline
glab ci retry --pipeline-id PIPELINE_ID 2>/dev/null || echo "Cannot retry pipeline"

# Run new pipeline on branch
glab ci run --branch BRANCH_NAME 2>/dev/null || echo "Cannot trigger pipeline"
```

### Artifact Management

```bash
# Download artifacts from latest pipeline (updated syntax)
glab job artifact <refName> <jobName> 2>/dev/null || echo "No artifacts available"

# Example: Download artifacts from specific branch and job
glab job artifact develop test-job 2>/dev/null || echo "No artifacts for develop/test-job"
```

## Configuration & Validation

### CI Configuration Analysis

```bash
# Validate .gitlab-ci.yml
glab ci lint 2>/dev/null || echo "CI configuration invalid"

# View CI configuration
glab ci config view 2>/dev/null || echo "No CI config found"
```

## Composite Analysis Commands

### Complete Pipeline Status Check

```bash
# Comprehensive pipeline analysis
PIPELINE_INFO=$(glab ci get --output json --branch develop 2>/dev/null)
if [ $? -eq 0 ] && [ "$PIPELINE_INFO" != "" ]; then
  echo "Pipeline Status:"
  echo "$PIPELINE_INFO" | jq -r '"ID: \(.id) | Status: \(.status) | Branch: \(.ref) | Duration: \(.duration)s | URL: \(.web_url)"'
  echo "Jobs:"
  echo "$PIPELINE_INFO" | jq -r '.jobs[] | "  \(.name): \(.status)"'
else
  echo "No pipeline information available"
fi
```

### Failed Job Analysis

```bash
# Find and display logs for failed jobs
PIPELINE_INFO=$(glab ci get --output json 2>/dev/null)
if [ $? -eq 0 ] && [ "$PIPELINE_INFO" != "" ]; then
  FAILED_JOBS=$(echo "$PIPELINE_INFO" | jq -r '.jobs[] | select(.status == "failed") | .id' 2>/dev/null)
  if [ "$FAILED_JOBS" != "" ]; then
    for job_id in $FAILED_JOBS; do
      echo "=== Logs for failed job $job_id ==="
      glab ci trace $job_id 2>/dev/null | tail -50 || echo "No logs available"
      echo ""
    done
  else
    echo "No failed jobs found"
  fi
else
  echo "No pipeline information available"
fi
```

### Continuous Pipeline Monitoring

```bash
# Monitor pipeline progress (for automated watching)
while true; do
  STATUS=$(glab ci get --output json 2>/dev/null | jq -r '.status // "unknown"' 2>/dev/null)
  if [ "$STATUS" = "running" ]; then
    echo "$(date): Pipeline still running..."
    glab ci get --output json 2>/dev/null | jq -r '.jobs[] | "  \(.name): \(.status)"' 2>/dev/null
    sleep 30
  else
    echo "$(date): Pipeline completed with status: $STATUS"
    break
  fi
done
```

## Error Handling Patterns

### Standard Error Handling Template

```bash
RESULT=$(command 2>/dev/null)
if [ $? -eq 0 ] && [ "$RESULT" != "" ]; then
  # Process successful result
  echo "$RESULT" | jq -r '.field'
else
  # Handle failure case
  echo "Command failed or no data available"
fi
```

### Troubleshooting Commands

```bash
# Check authentication
glab auth status

# Check project access
glab repo view --output json | jq -r '.permissions // "No permissions info"'

# Check if pipelines exist
glab ci get --output json 2>/dev/null | jq -r 'keys[]' 2>/dev/null || echo "No pipeline data structure available"
```

## Environment Variables

Set these for consistent behavior:

```bash
export GLAB_CONFIG_FILE="/path/to/config.yml"
export GITLAB_TOKEN="your_token_here"
export GITLAB_HOST="gitlab.com"
```

## Quick Reference Commands

### Most Commonly Used

```bash
# Current pipeline status
glab ci get --output json 2>/dev/null | jq -r '"\(.status) | \(.ref) | \(.web_url)"' 2>/dev/null

# Job count by status
glab ci get --output json 2>/dev/null | jq -r '.jobs | group_by(.status) | map("\(.[0].status): \(length)") | .[]' 2>/dev/null

# Latest pipeline duration
glab ci get --output json 2>/dev/null | jq -r '.duration // "unknown"' 2>/dev/null

# Failed job names
glab ci get --output json 2>/dev/null | jq -r '.jobs[] | select(.status == "failed") | .name' 2>/dev/null
```

## Direct API Fallback Methods

When glab commands fail (which happens frequently), use these direct API methods:

```bash
# Setup API variables
setup_gitlab_api() {
  export GITLAB_HOST="${GITLAB_HOST:-https://gitlab.com}"
  export GITLAB_TOKEN="${GITLAB_TOKEN:-$(glab auth token 2>/dev/null)}"
  export CI_PROJECT_ID="${CI_PROJECT_ID:-$(glab repo view --output json 2>/dev/null | jq -r '.id // ""')}"

  if [[ -z "$GITLAB_TOKEN" ]]; then
    echo "❌ No GitLab token available. Set GITLAB_TOKEN environment variable."
    return 1
  fi
}

# Get pipeline details via API
get_pipeline_via_api() {
  local pipeline_id="$1"
  setup_gitlab_api || return 1

  curl -s -H "Authorization: Bearer $GITLAB_TOKEN" \
    "$GITLAB_HOST/api/v4/projects/$CI_PROJECT_ID/pipelines/$pipeline_id" | jq '.'
}

# Get job logs via API
get_job_logs_via_api() {
  local job_id="$1"
  setup_gitlab_api || return 1

  curl -s -H "Authorization: Bearer $GITLAB_TOKEN" \
    "$GITLAB_HOST/api/v4/projects/$CI_PROJECT_ID/jobs/$job_id/trace"
}

# List pipelines via API
list_pipelines_via_api() {
  local branch="${1:-}"
  setup_gitlab_api || return 1

  local url="$GITLAB_HOST/api/v4/projects/$CI_PROJECT_ID/pipelines"
  if [[ -n "$branch" ]]; then
    url="$url?ref=$branch"
  fi

  curl -s -H "Authorization: Bearer $GITLAB_TOKEN" "$url" | jq -r '.[] | "\(.id) \(.status) \(.ref) \(.created_at)"'
}

# Retry pipeline via API
retry_pipeline_via_api() {
  local pipeline_id="$1"
  setup_gitlab_api || return 1

  curl -X POST -H "Authorization: Bearer $GITLAB_TOKEN" \
    "$GITLAB_HOST/api/v4/projects/$CI_PROJECT_ID/pipelines/$pipeline_id/retry" | jq '.'
}

# Cancel pipeline via API
cancel_pipeline_via_api() {
  local pipeline_id="$1"
  setup_gitlab_api || return 1

  curl -X POST -H "Authorization: Bearer $GITLAB_TOKEN" \
    "$GITLAB_HOST/api/v4/projects/$CI_PROJECT_ID/pipelines/$pipeline_id/cancel" | jq '.'
}
```

## Commands to Avoid in Automation (Updated Based on Production Feedback)

### ❌ Commands That Consistently Fail:

```bash
# These commands fail with 403/404 errors even with valid authentication:
glab ci view <pipeline_id>        # 404 Not Found - Use API fallback instead
glab ci get <pipeline_id> --output json  # "accepts 0 arg(s), received 1"
glab ci trace <job_name>          # Only works with numeric IDs, not names!
glab ci list --limit <number>     # Unknown flag - use 'head' instead
glab ci trace --job <job_name>    # Unknown flag
glab job list                     # Often fails
glab pipeline list                # Often fails
```

### ✅ Commands That Work Reliably:

```bash
glab auth status                  # Authentication check
glab ci list                      # Basic pipeline list (but no --limit flag)
glab ci status                    # Current branch status
glab ci status --branch <branch>  # Specific branch status
glab ci get --output json         # Latest pipeline (no pipeline ID argument)
glab ci trace <numeric_job_id>    # Works ONLY with numeric IDs
glab api <endpoint>               # Direct API access
glab ci lint                      # Validate CI config
```

## Notes

### Critical Updates (Based on Production Feedback):

- **Job Tracing**: `glab ci trace` ONLY works with numeric job IDs, NOT job names
- **Pipeline Details**: `glab ci view` consistently fails with 404 errors - use API fallback
- **Permission Issues**: Many detailed operations fail even with valid authentication
- **Fallback Strategy**: Always implement cascading fallbacks for critical operations
- **Debug Mode**: Use `GITLAB_DEBUG=true` to troubleshoot command failures

### Best Practices:

- The `glab ci get` command is the most reliable for current pipeline information
- Always use `--output json` with `jq` for parsing structured data
- Always validate permissions before attempting detailed operations
- Always include error handling with `|| echo "fallback message"`
- All commands are designed for non-interactive automation environments
- When glab fails, fall back to direct API calls using curl