playwright-testrail-sync/README.md

TestRail Integration for Playwright with comprehensive logging and error handling

# playwright-testrail-sync

[![npm version](https://img.shields.io/npm/v/playwright-testrail-sync)](https://www.npmjs.com/package/playwright-testrail-sync)
[![MIT License](https://img.shields.io/badge/license-MIT-green.svg)](./LICENSE)

TypeScript integration to sync Playwright test results with TestRail, featuring automatic screenshot and video uploads, flexible configuration, comprehensive error handling, and clean logs. Automatically manages test suites, test cases, and sections in TestRail. Features smart test case matching, auto-creation of missing components, and seamless test result synchronization.

**✨ What happens automatically:**
- 🔍 **Finds or creates** TestRail test suites
- 🎯 **Matches existing test cases** using smart fuzzy matching
- 📁 **Creates sections** based on your test file folders
- ➕ **Creates missing test cases** when needed
- 🔗 **Maps Playwright tests** to TestRail cases

## Quick Start

Get up and running in 3 steps:

1. **Install the package:**
```bash
npm install playwright-testrail-sync
```

2. **Create a `.env` file:**
```bash
TESTRAIL_HOST=https://yourcompany.testrail.io
TESTRAIL_USERNAME=your-email@company.com
TESTRAIL_API_KEY=your-api-key-here
TESTRAIL_PROJECT_ID=123
```

3. **Add to your `playwright.config.ts`:**
```typescript
export default defineConfig({
  reporter: [
    ['html'],
    ['playwright-testrail-sync'], // Uses .env variables automatically
  ],
})
```

🎉 That’s it — run your Playwright tests, and results will sync to TestRail automatically.

## Prerequisites

Before getting started, ensure you have the following:

### 🔑 **TestRail Access**
- Active TestRail account with API access
- TestRail project ID
- API key generated from [TestRail settings](#-getting-your-testrail-api-key)

### ⚙️ **Environment Setup**

#### **Using .env Files (Recommended)**

**Step 1: Create a `.env` file in your project root (or add to existing one) the following variables:**

For most projects, setting up these 4 variables is enough to get started:

```bash
# .env - TestRail Configuration
TESTRAIL_HOST=https://yourcompany.testrail.io
TESTRAIL_USERNAME=your-email@company.com
TESTRAIL_API_KEY=your-api-key-here
TESTRAIL_PROJECT_ID=123
```

**Step 2: Configure your project to load .env variables:**

**If needed - configure your `playwright.config.ts` to be able to use the .env:**

```javascript
// Example: In your playwright.config.ts
import dotenv from 'dotenv'
import path from 'path'
import { fileURLToPath } from 'url'

const dirname = path.dirname(fileURLToPath(import.meta.url))
dotenv.config({ path: path.resolve(dirname, '../your-app/.env') }) // Adjust path as needed

export default defineConfig({
  reporter: [
    ['html'],
    ['playwright-testrail-sync'], // Uses .env variables automatically
  ],
})
```


**For more advanced setup, please check the [Programmatic Configuration](#programmatic-configuration) section below for extensive configuration options including logging, attachments, validation, and more.**

#### **Complete .env Example**

Here's a comprehensive `.env` file with all available options:

```bash
# ===========================================
# TestRail Configuration (.env file)
# ===========================================

# Required: TestRail instance URL
TESTRAIL_HOST=https://yourcompany.testrail.io

# Required: Your TestRail email address  
TESTRAIL_USERNAME=your-email@company.com

# Required: TestRail API key (recommended) or password (deprecated)
TESTRAIL_API_KEY=your-api-key-here
# TESTRAIL_PASSWORD=your-password-here  # Deprecated, use TESTRAIL_API_KEY instead

# Required: TestRail project ID
# Find this in your TestRail URL: /index.php?/projects/overview/[PROJECT_ID]
TESTRAIL_PROJECT_ID=123

# Optional: TestRail suite ID (if you want to limit to specific suite)
# Find this in your TestRail URL: /index.php?/suites/view/[SUITE_ID]
TESTRAIL_SUITE_ID=456

# Optional: Test run name (if not specified, will use default one: Playwright Test Runf)
TESTRAIL_RUN_NAME=Your Custom Test Run Name

# Optional: Existing test run ID (if you want to add results to existing run. Make sure that your Test Run Name keeps the same)
TESTRAIL_RUN_ID=789

# ===========================================
# Attachment Configuration
# ===========================================

# Optional: Enable/disable video uploads (default: false)
TESTRAIL_UPLOAD_VIDEOS=true

# Optional: Maximum attachment size in MB (default: 10)
TESTRAIL_MAX_ATTACHMENT_SIZE=10

# Optional: Enable/disable attachments (default: true)
TESTRAIL_ATTACHMENTS_ENABLED=true

# Optional: Take screenshots on test failure (default: true)
TESTRAIL_SCREENSHOT_ON_FAILURE=true

# ===========================================
# Logging Configuration
# ===========================================

# Optional: Log level (debug, info, warn, error)
TESTRAIL_LOG_LEVEL=info

# Optional: Enable verbose mode
TESTRAIL_VERBOSE_LOGGING=true
```

### Programmatic Configuration

**Use this approach for library-like setups, CI pipelines where .env isn't available, or when you need more control over configuration:**

**Extensive configuration options:**

```typescript
// playwright.config.ts
export default defineConfig({
  reporter: [
    ['html'],
    ['playwright-testrail-sync', {
      // TestRail connection settings
      testRail: {
        host: 'https://your-instance.testrail.io',
        username: 'your-email@company.com',
        apiKey: 'your-api-key',
        projectId: 1,
        suiteId: 1, // Optional, otherwise will create a new SuiteId with the name: Test Suite generated by Playwright
        testRunName: 'Your Custom Test Run Name', // Optional, otherwise uses a default name: Playwright Test Run - DD/MM/YYYY 23:59:59
        runId: 12345, // Optional: use existing run
        assignedToId: 1, // Optional: assign to user
        milestoneId: 1, // Optional: associate with milestone
        refs: 'v1.2.3', // Optional: test run references
      },
      
      // Logging configuration
      logging: {
        level: 'info', // debug, info, warn, error
        verbose: true,
      },
      
      // Attachment settings
      attachments: {
        enabled: true,
        maxSize: 10, // MB
        screenshotOnFailure: true,
        uploadVideos: false, // Default: false (performance/storage optimization)
        // allowedTypes: ['png', 'jpg', 'mp4', 'webm'] // Optional: restrict file types
        // By default, all file types are allowed. Only specify allowedTypes to restrict uploads.
      },
      
      
      // Validation settings
      validation: {
        strictMode: true,
        validateCaseIds: true,
        validateRunId: true
      }
    }]
  ]
})
```

### 🔄 **Configuration Priority**

**Environment variables always take precedence** over configuration file settings:

```bash
# Environment variables override config file
export TESTRAIL_API_KEY="your-api-key"
export TESTRAIL_VERBOSE_LOGGING="true"
```

This allows you to:

- **Override secrets** without changing code
- **Use different settings** per environment  
- **Keep sensitive data** in environment variables


## 🔑 Getting Your TestRail API Key

To get your TestRail API key, follow these steps:

1. **Log into your TestRail instance** (e.g., [https://yourCompany.testrail.io](https://yourCompany.testrail.io))
2. **Navigate to My Settings**:
   - Click on your profile/avatar in the top right corner
   - Select "My Settings" from the dropdown menu
3. **Go to API Keys section**:
   - In the left sidebar, click on "API Keys"
4. **Create a new API key**:
   - Click "Add Key" button
   - Give your key a descriptive name (e.g., "Playwright Integration")
   - Click "Add" to generate the key

```bash
# Store your API key in environment variables
export TESTRAIL_API_KEY="your-api-key-here"
```


## Output Examples

### Successful Test Run

```
[2025-10-04 22:35:12] [Playwright-TestRail-Sync] info: Test execution started 
[2025-10-04 22:35:13] [Playwright-TestRail-Sync] info: Creating new TestRail run... 
[2025-10-04 22:35:15] [Playwright-TestRail-Sync] info: Created new Test Suite: Test Suite generated by Playwright (ID: 1234) 
[2025-10-04 22:35:18] [Playwright-TestRail-Sync] info: Created new TestRail Run: 12345
[2025-10-04 22:35:45] [Playwright-TestRail-Sync] info: Test passed: User should be able to login 
[2025-10-04 22:38:26] [Playwright-TestRail-Sync] info: TestRail sync completed 
[2025-10-04 22:38:26] [Playwright-TestRail-Sync] info: testRailUrl: https://your-instance.testrail.io/index.php?/runs/view/12345 
```

## 🧠 **How Automatic Management Works**

### **🔄 Automatic Test Suite Detection**

```typescript
// The package automatically:
// 1. Looks for existing test suites by name
// 2. Creates new suites if none exist
// To keep your custom Test Suite name in TestRail:
// - Use the TESTRAIL_SUITE_ID environment variable, or
// - Set the "suiteId" in playwright.config
```

### **🎯 Smart Test Case Matching**

```typescript
// Test case matching strategies (in order):
// 1. Exact match: "User Login" === "User Login"
// 2. Normalized match: "User-Login" === "User Login" 
// 3. Fuzzy match: "User Login Test" ≈ "User Login" (90%+ similarity)
// 4. Partial match: "Login" contained in "User Login Test"
```

### **📁 Folder-Based Section Creation**

```typescript
// Your test structure:
tests/
  auth/
    login.test.ts     → Creates "Auth" section
  checkout/
    payment.test.ts   → Creates "Checkout" section
  api/
    users.test.ts     → Creates "Api" section
```

## Error Handling

The integration provides comprehensive error handling:

- **Invalid TestRail credentials** → Clear error messages with setup instructions
- **Missing test case IDs** → Warnings for tests without case IDs
- **Network failures** → Clear error messages with actionable guidance
- **Invalid case IDs** → Validation before upload attempts
- **Attachment failures** → Graceful degradation without stopping tests

## Troubleshooting

### Common Issues

**Issue**: `Configuration validation failed: TESTRAIL_HOST is required`
**Solution**: Ensure all required environment variables are set

**Issue**: `Failed to upload results: Invalid status`
**Solution**: Check that your TestRail instance supports the status being used

### Debug Mode

Enable verbose logging to troubleshoot issues:

```bash
TESTRAIL_VERBOSE_LOGGING=true npx playwright test
```

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.