playwright-testrail-sync/README.md

TestRail Integration for Playwright with comprehensive logging and error handling

# playwright-testrail-sync

TypeScript integration to sync Playwright test results with TestRail, featuring automatic screenshot and ideo uploads, flexible configuration, comprehensive error handling, and clean logs.

## Installation

```bash
# Using npm
npm install playwright-testrail-sync

# Using yarn
yarn add playwright-testrail-sync
```

## Prerequisites

Before getting started, ensure you have the following:

### 🔑 **TestRail Access**
- Active TestRail account with API access
- TestRail project and suite IDs
- API key generated from TestRail settings

### ⚙️ **Environment Setup**
Configure these environment variables in your system or `.env` file:

| Variable | Required | Description | Example |
|----------|----------|-------------|---------|
| `TESTRAIL_HOST` | ✅ | Your TestRail instance URL | `https://yourcompany.testrail.io` |
| `TESTRAIL_USERNAME` | ✅ | TestRail email address | `your-email@company.com` |
| `TESTRAIL_PASSWORD` | ✅ | TestRail API key | `abc123def456` |
| `TESTRAIL_PROJECT_ID` | ✅ | TestRail project ID | `1` |
| `TESTRAIL_SUITE_ID` | ✅ | TestRail suite ID | `1` |
| `TESTRAIL_RUN_NAME` | ❌ | Test run name template | `"Playwright Test Run"` |
| `TESTRAIL_RUN_ID` | ❌ | Use existing run (optional) | `12345` |
| `TESTRAIL_VERBOSE_LOGGING` | ❌ | Enable debug logging | `true` |
| `TESTRAIL_SCREENSHOT_ON_FAILURE` | ❌ | Auto-screenshot on test failure | `true` |
| `TESTRAIL_UPLOAD_VIDEOS` | ❌ | Upload test videos to TestRail | `false` |

> **💡 Advanced Configuration**: For more detailed settings like logging levels, attachment settings, and validation modes, use the [Programmatic Configuration](#programmatic-configuration) in your `playwright.config.ts` file.

### 📋 **Getting Your TestRail IDs**

1. **Project ID**: Found in your TestRail URL: `/index.php?/projects/overview/1` → ID is `1`
2. **Suite ID**: Navigate to your project → Test Suites → Click on suite → URL shows the ID
3. **API Key**: Go to TestRail → My Settings → API Keys → Generate new key

## Quick Start

### 1. Configure Playwright

Add the TestRail integration to your `playwright.config.ts`:

```typescript
import { defineConfig } from '@playwright/test'

export default defineConfig({
  reporter: [
    ['html'],
    ['playwright-testrail-sync']
  ],
  // ... your other config
})
```

### 2. Add Test Case IDs

Include TestRail case IDs in your test titles using the `C12345` format:

```typescript
// Single case ID
test('C12345: User login with valid credentials', async ({ page }) => {
  // Your test code
})

// Multiple case IDs
test('C12345 C12346 C12347: Complex user workflow', async ({ page }) => {
  // Your test code
})
```

### 3. Run Your Tests

Execute your tests and watch the magic happen:

```bash
# Using npm
npx playwright test

# Using yarn
yarn playwright test
```

## Features

### 🚀 **Automatic Test Run Management**
- Creates new TestRail runs automatically
- Links test results to existing runs
- Handles run naming with timestamps

### 📊 **Smart Test Case Matching**
- Extracts case IDs from test titles (`C12345` format)
- Supports multiple case IDs per test
- Validates case existence before upload

### 📸 **Rich Media Support**
- Automatic screenshot capture on failures
- Video recording integration
- Log file attachments
- Custom file uploads

### 🔍 **Advanced Logging**
- Clean, structured log output
- Verbose mode for debugging
- Error tracking and reporting
- Performance metrics

### ⚙️ **Flexible Configuration**
- Environment variable configuration
- Programmatic options
- Custom run naming
- Attachment filtering

## Configuration Options


### Programmatic Configuration

```typescript
// playwright.config.ts
export default defineConfig({
  reporter: [
    ['html'],
    ['playwright-testrail-sync', {
      // TestRail connection settings
      testRail: {
        host: 'https://your-instance.testrail.io',
        username: 'your-email@company.com',
        password: 'your-api-key',
        projectId: 1,
        suiteId: 1,
        testRunName: 'My Test Run', // Optional, otherwise uses a default name: Test Run - Date/Time stamp
        runId: 12345, // Optional: use existing run
        assignedToId: 1, // Optional: assign to user
        milestoneId: 1, // Optional: associate with milestone
        refs: 'v1.2.3', // Optional: test run references
      },
      
      // Logging configuration
      logging: {
        level: 'info', // debug, info, warn, error
        verbose: true,
      },
      
      // Attachment settings
      attachments: {
        enabled: true,
        maxSize: 10, // MB
        screenshotOnFailure: true,
        uploadVideos: true,
        allowedTypes: ['png', 'jpg', 'mp4', 'webm']
      },
      
      
      // Validation settings
      validation: {
        strictMode: true,
        validateCaseIds: true,
        validateRunId: true
      }
    }]
  ]
})
```

### 🔄 **Configuration Priority**

**Environment variables always take precedence** over configuration file settings:

```bash
# Environment variables override config file
export TESTRAIL_PASSWORD="different-password"
export TESTRAIL_VERBOSE_LOGGING="true"
```

This allows you to:
- **Override secrets** without changing code
- **Use different settings** per environment  
- **Keep sensitive data** in environment variables

## Test Case ID Format

The integration recognizes TestRail case IDs in test titles using the `C` prefix:

```typescript
// ✅ Valid formats
test('C12345: Test description', () => {})
test('C12345 C12346: Multiple cases', () => {})
test('Login C12345 should work', () => {})
test('C12345-C12346-C12347: Complex test', () => {})

// ❌ Invalid formats
test('12345: Missing C prefix', () => {})
test('TC12345: Wrong prefix', () => {})
```

## Output Examples

### Successful Test Run

```
[Playwright-TestRail-Sync] info: Test execution started 
[Playwright-TestRail-Sync] info: Creating new TestRail run... 
[Playwright-TestRail-Sync] info: Created new TestRail run: 12345 
[Playwright-TestRail-Sync] info: Test passed: C12345 should login with valid credentials 
[Playwright-TestRail-Sync] info: Test passed: C12346 should display user dashboard
[Playwright-TestRail-Sync] info: Test passed: C12347 should logout successfully  
[Playwright-TestRail-Sync] info: TestRail sync completed 
[Playwright-TestRail-Sync] info: testRailUrl: https://your-instance.testrail.io/index.php?/runs/view/12345 
```

## Error Handling

The integration provides comprehensive error handling:

- **Invalid TestRail credentials** → Clear error messages with setup instructions
- **Missing test case IDs** → Warnings for tests without case IDs
- **Network failures** → Clear error messages with actionable guidance
- **Invalid case IDs** → Validation before upload attempts
- **Attachment failures** → Graceful degradation without stopping tests

## Troubleshooting

### Common Issues

**Issue**: `Configuration validation failed: TESTRAIL_HOST is required`
**Solution**: Ensure all required environment variables are set

**Issue**: `TestRail case not found: C12345`
**Solution**: Verify the case ID exists in your TestRail project

**Issue**: `Failed to upload results: Invalid status`
**Solution**: Check that your TestRail instance supports the status being used

### Debug Mode

Enable verbose logging to troubleshoot issues:

```bash
TESTRAIL_VERBOSE_LOGGING=true npx playwright test
```

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.