@applitools/req/README.md

Applitools fetch-based request library

# @applitools/req

A powerful, flexible HTTP request library with advanced features like retry logic, hooks, fallbacks, and timeout management.

## Table of Contents

- [Installation](#installation)
- [Basic Usage](#basic-usage)
- [API Reference](#api-reference)
  - [req()](#req)
  - [makeReq()](#makereq)
  - [Options](#options)
- [Advanced Features](#advanced-features)
  - [Retry Logic](#retry-logic)
  - [Hooks](#hooks)
  - [Fallbacks](#fallbacks)
  - [Timeouts](#timeouts)
- [Chaining Options](#chaining-options)
- [Examples](#examples)

## Installation

```bash
yarn add @applitools/req
```

## Basic Usage

```typescript
import {req} from '@applitools/req'

// Simple GET request
const response = await req('https://api.example.com/data')
const data = await response.json()
```

## API Reference

### req()

The main function for making HTTP requests.

**Signature:**
```typescript
req(input: string | URL | Request, ...options: Options[]): Promise<Response>
```

**Parameters:**
- `input` - URL string, URL object, or Request object
- `options` - One or more option objects that will be merged (optional)

**Returns:** Promise that resolves to a Response object

### makeReq()

Creates a req function with predefined base options.

**Signature:**
```typescript
makeReq<TOptions>(baseOptions: Partial<TOptions>): Req<TOptions>
```

**Example:**
```typescript
const apiReq = makeReq({
  baseUrl: 'https://api.example.com',
  headers: {'Authorization': 'Bearer token123'}
})

// All requests will use the base options
const response = await apiReq('/users')
```

### Options

All available options for configuring requests:

#### `baseUrl`
**Type:** `string`

Base URL for relative paths. Automatically adds trailing slash if missing.

**Example:**
```typescript
await req('./users', {baseUrl: 'https://api.example.com/v1'})
// Makes request to: https://api.example.com/v1/users
```

#### `method`
**Type:** `string`

HTTP method (uppercase). Overrides method from Request object.

**Example:**
```typescript
await req('https://api.example.com/users', {method: 'POST'})
```

#### `query`
**Type:** `Record<string, string | boolean | number | undefined>`

Query parameters to append to URL. Merges with existing query params. Undefined values are ignored.

**Example:**
```typescript
await req('https://api.example.com/search?page=1', {
  query: {
    limit: 10,
    filter: 'active',
    skip: undefined  // This won't be added
  }
})
// URL: https://api.example.com/search?page=1&limit=10&filter=active
```

#### `headers`
**Type:** `Record<string, string | string[] | undefined>`

HTTP headers. Merges with headers from Request object. Undefined values are filtered out.

**Example:**
```typescript
await req('https://api.example.com/data', {
  headers: {
    'Authorization': 'Bearer token',
    'Content-Type': 'application/json',
    'X-Optional': undefined  // Won't be sent
  }
})
```

#### `body`
**Type:** `NodeJS.ReadableStream | ArrayBufferView | string | Record<string, any> | any[] | null`

Request body. Plain objects and arrays are automatically serialized to JSON with appropriate content-type header.

**Example:**
```typescript
// Automatic JSON serialization
await req('https://api.example.com/users', {
  method: 'POST',
  body: {name: 'John', age: 30}
})

// Binary data
await req('https://api.example.com/upload', {
  method: 'POST',
  body: Buffer.from('binary data')
})
```

#### `proxy`
**Type:** `Proxy | ((url: URL) => Proxy | undefined)`

Proxy configuration. Can be an object or function that returns proxy settings based on URL.

**Example:**
```typescript
// Static proxy
await req('https://api.example.com/data', {
  proxy: {
    url: 'http://proxy.company.com:8080',
    username: 'user',
    password: 'pass'
  }
})

// Dynamic proxy
await req('https://api.example.com/data', {
  proxy: (url) => {
    if (url.hostname.includes('internal')) {
      return {url: 'http://internal-proxy:8080'}
    }
  }
})
```

#### `useDnsCache`
**Type:** `boolean`

Enable DNS caching for improved performance.

**Example:**
```typescript
await req('https://api.example.com/data', {useDnsCache: true})
```

#### `connectionTimeout`
**Type:** `number`

Total timeout in milliseconds for the entire connection including all retries. Once exceeded, throws `ConnectionTimeoutError`.

**Example:**
```typescript
await req('https://api.example.com/data', {
  connectionTimeout: 30000,  // 30 seconds total
  retry: {statuses: [500]}
})
```

#### `requestTimeout`
**Type:** `number | {base: number; perByte: number}`

Timeout for a single request in milliseconds. Can be dynamic based on request body size.

**Example:**
```typescript
// Fixed timeout
await req('https://api.example.com/data', {
  requestTimeout: 5000  // 5 seconds per request
})

// Dynamic timeout based on body size
await req('https://api.example.com/upload', {
  method: 'POST',
  body: largeBuffer,
  requestTimeout: {
    base: 1000,      // 1 second base
    perByte: 0.001   // + 1ms per byte
  }
})
```

#### `retryTimeout`
**Type:** `number`

Maximum duration in milliseconds for all retry attempts across all retry strategies. Once exceeded, throws `RetryTimeoutError`.

**Example:**
```typescript
await req('https://api.example.com/data', {
  retryTimeout: 30000,  // Stop all retries after 30 seconds total
  retry: [
    {statuses: [500], timeout: 1000},
    {codes: ['ECONNRESET'], timeout: 500}
  ]
})
```

#### `retry`
**Type:** `Retry | Retry[]`

Retry configuration(s). Multiple retry configs can be provided as array.

See [Retry Logic](#retry-logic) for detailed examples.

#### `hooks`
**Type:** `Hooks | Hooks[]`

Lifecycle hooks for request interception and modification.

See [Hooks](#hooks) for detailed examples.

#### `fallbacks`
**Type:** `Fallback | Fallback[]`

Fallback strategies for handling failures.

See [Fallbacks](#fallbacks) for detailed examples.

#### `keepAliveOptions`
**Type:** `{keepAlive: boolean; keepAliveMsecs?: number}`

HTTP agent keep-alive configuration.

**Example:**
```typescript
await req('https://api.example.com/data', {
  keepAliveOptions: {
    keepAlive: true,
    keepAliveMsecs: 1000
  }
})
```

#### `signal`
**Type:** `AbortSignal`

Abort signal for canceling requests.

**Example:**
```typescript
const controller = new AbortController()

// Cancel after 5 seconds
setTimeout(() => controller.abort(), 5000)

await req('https://api.example.com/data', {
  signal: controller.signal
})
```

## Advanced Features

### Retry Logic

The `retry` option enables automatic retrying of failed requests based on various conditions.

#### Retry Configuration

```typescript
interface Retry {
  limit?: number                    // Max retry attempts (default: unlimited if undefined)
  timeout?: number | number[]       // Delay between retries in ms (default: 0 - no delay)
  statuses?: number[]               // HTTP status codes to retry (default: none)
  codes?: string[]                  // Error codes to retry (default: none)
  validate?: (options) => boolean   // Custom validation function (default: undefined)
}
```

**Default Behavior:**
- **`limit`**: If undefined or not set, retries will continue indefinitely (use with caution - combine with `connectionTimeout` or `retryTimeout`)
- **`timeout`**: If undefined, retries happen immediately with no delay (0ms)
- **`statuses`**: If undefined, no status codes trigger retries (must be explicitly configured)
- **`codes`**: If undefined, no error codes trigger retries (must be explicitly configured)
- **`validate`**: If undefined, only `statuses` and `codes` are checked
- **Retry trigger**: At least one of `statuses`, `codes`, or `validate` must match for a retry to occur
- **`Retry-After` header**: If present in the response, overrides the configured `timeout` value

#### Examples

**Retry on specific status codes:**
```typescript
await req('https://api.example.com/data', {
  retry: {
    statuses: [500, 502, 503],  // Retry on server errors
    limit: 3,                    // Max 3 attempts
    timeout: 1000                // Wait 1 second between retries
  }
})
```

**Retry on network errors:**
```typescript
await req('https://api.example.com/data', {
  retry: {
    codes: ['ECONNRESET', 'ETIMEDOUT'],
    limit: 5,
    timeout: 2000
  }
})
```

**Exponential backoff:**
```typescript
await req('https://api.example.com/data', {
  retry: {
    statuses: [429, 500],
    limit: 4,
    timeout: [1000, 2000, 4000, 8000]  // Double delay each time
  }
})
```

**Custom validation:**
```typescript
await req('https://api.example.com/data', {
  retry: {
    validate: async ({response, error}) => {
      if (error) return true
      if (response?.status === 429) {
        // Check rate limit header
        return response.headers.has('Retry-After')
      }
      return false
    },
    limit: 3
  }
})
```

**Multiple retry strategies:**
```typescript
await req('https://api.example.com/data', {
  retry: [
    // Retry network errors quickly
    {codes: ['ECONNRESET'], limit: 3, timeout: 500},
    // Retry server errors with longer delay
    {statuses: [500, 503], limit: 2, timeout: 2000}
  ]
})
```

**Retry with timeout limit:**
```typescript
await req('https://api.example.com/data', {
  retryTimeout: 10000,  // Stop all retries after 10 seconds
  retry: [
    {codes: ['ECONNRESET'], timeout: 500},
    {statuses: [500, 503], timeout: 1000}
  ]
})
// If retries take longer than 10 seconds total, RetryTimeoutError is thrown
```

### Hooks

Hooks allow you to intercept and modify requests/responses at various lifecycle stages.

#### Available Hooks

```typescript
interface Hooks {
  afterOptionsMerged?(options): TOptions | void
  beforeRequest?(options): Request | void
  beforeRetry?(options): Request | Stop | void
  afterResponse?(options): Response | void
  afterError?(options): Error | void
  unknownBodyType?(options): void
}
```

#### Examples

**Add authentication header:**
```typescript
await req('https://api.example.com/data', {
  hooks: {
    beforeRequest: ({request}) => {
      request.headers.set('Authorization', `Bearer ${getToken()}`)
    }
  }
})
```

**Log all requests:**
```typescript
await req('https://api.example.com/data', {
  hooks: {
    beforeRequest: ({request}) => {
      console.log(`${request.method} ${request.url}`)
    },
    afterResponse: ({response}) => {
      console.log(`Response: ${response.status}`)
    }
  }
})
```

**Conditional retry prevention:**
```typescript
import {stop} from '@applitools/req'

await req('https://api.example.com/data', {
  retry: {statuses: [500]},
  hooks: {
    beforeRetry: async ({response, attempt, stop}) => {
      const data = await response?.json()
      if (data?.error === 'FATAL') {
        console.log('Fatal error, stopping retries')
        return stop
      }
      console.log(`Retry attempt ${attempt}`)
    }
  }
})
```

**Transform response:**
```typescript
await req('https://api.example.com/data', {
  hooks: {
    afterResponse: async ({response}) => {
      if (!response.ok) {
        const error = await response.text()
        throw new Error(`API Error: ${error}`)
      }
    }
  }
})
```

**Modify request before retry:**
```typescript
await req('https://api.example.com/data', {
  retry: {statuses: [401]},
  hooks: {
    beforeRetry: async ({request, attempt}) => {
      // Refresh token on 401
      const newToken = await refreshAuthToken()
      request.headers.set('Authorization', `Bearer ${newToken}`)
      return request
    }
  }
})
```

**Multiple hooks:**
```typescript
await req('https://api.example.com/data', {
  hooks: [
    {
      beforeRequest: ({request}) => {
        request.headers.set('X-Request-ID', generateId())
      }
    },
    {
      beforeRequest: ({request}) => {
        request.headers.set('X-Timestamp', Date.now().toString())
      }
    }
  ]
})
```

### Fallbacks

Fallbacks provide alternative strategies when requests fail.

#### Fallback Configuration

```typescript
interface Fallback {
  shouldFallbackCondition: (options) => boolean | Promise<boolean>
  updateOptions?: (options) => Options | Promise<Options>
  cache?: Map<string, boolean>
}
```

#### Examples

**Fallback to different endpoint:**
```typescript
await req('https://api-primary.example.com/data', {
  fallbacks: {
    shouldFallbackCondition: ({response}) => response.status >= 500,
    updateOptions: ({options}) => ({
      ...options,
      baseUrl: 'https://api-backup.example.com'
    })
  }
})
```

**Try with authentication if unauthorized:**
```typescript
await req('https://api.example.com/data', {
  fallbacks: {
    shouldFallbackCondition: ({response}) => response.status === 401,
    updateOptions: async ({options}) => ({
      ...options,
      headers: {
        ...options.headers,
        'Authorization': `Bearer ${await getToken()}`
      }
    })
  }
})
```

**Multiple fallback strategies:**
```typescript
await req('https://api.example.com/data', {
  fallbacks: [
    // First try: enable keep-alive
    {
      shouldFallbackCondition: ({response}) => response.status === 503,
      updateOptions: ({options}) => ({
        ...options,
        keepAliveOptions: {keepAlive: true}
      })
    },
    // Second try: use proxy
    {
      shouldFallbackCondition: ({response}) => response.status === 403,
      updateOptions: ({options}) => ({
        ...options,
        proxy: {url: 'http://proxy.example.com:8080'}
      })
    }
  ]
})
```

### Timeouts

Three types of timeouts control request timing:

#### `connectionTimeout`
Total timeout across all retries and delays. Throws `ConnectionTimeoutError` when exceeded.

```typescript
await req('https://api.example.com/data', {
  connectionTimeout: 30000,  // 30 seconds total for all attempts
  retry: {
    statuses: [500],
    limit: 5,
    timeout: 2000
  }
})
// If all 5 retries take too long, ConnectionTimeoutError is thrown
```

#### `requestTimeout`
Timeout for each individual request attempt. Throws `RequestTimeoutError` when exceeded.

```typescript
await req('https://api.example.com/data', {
  requestTimeout: 5000,      // Each attempt times out after 5 seconds
  retry: {
    codes: ['RequestTimeout'],
    limit: 3
  }
})
```

#### `retryTimeout`
Total timeout for all retry attempts across all retry strategies. Throws `RetryTimeoutError` when exceeded.

```typescript
await req('https://api.example.com/data', {
  retryTimeout: 15000,       // Stop retrying after 15 seconds total
  retry: [
    {codes: ['ECONNRESET'], timeout: 1000},
    {statuses: [500, 503], timeout: 2000}
  ]
})
// If retry process takes longer than 15 seconds, RetryTimeoutError is thrown
```

**Difference between timeouts:**
- `connectionTimeout`: Covers the entire request lifecycle including initial attempt and all retries
- `retryTimeout`: Only covers retry attempts (excludes the initial request)
- `requestTimeout`: Applies to each individual request attempt

#### Dynamic request timeout based on body size:
```typescript
await req('https://api.example.com/upload', {
  method: 'POST',
  body: fileBuffer,
  requestTimeout: {
    base: 5000,      // 5 seconds baseline
    perByte: 0.01    // + 10ms per byte
  }
})
// For 1MB file: timeout = 5000 + (1048576 * 0.01) ≈ 15.5 seconds
```

## Chaining Options

Multiple option objects can be passed and will be deeply merged. This enables powerful composition patterns.

### Merge Behavior

- Simple properties (strings, numbers) are overridden
- Objects (`query`, `headers`) are merged
- Arrays (`retry`, `hooks`, `fallbacks`) are concatenated
- Later options take precedence

### Examples

**Basic chaining:**
```typescript
const baseOptions = {
  baseUrl: 'https://api.example.com',
  headers: {'User-Agent': 'MyApp/1.0'}
}

const authOptions = {
  headers: {'Authorization': 'Bearer token'}
}

// Merged result:
// - baseUrl: 'https://api.example.com'
// - headers: {'User-Agent': 'MyApp/1.0', 'Authorization': 'Bearer token'}
await req('/users', baseOptions, authOptions)
```

**Override with precedence:**
```typescript
const options1 = {
  requestTimeout: 5000,
  headers: {'X-Version': '1'}
}

const options2 = {
  requestTimeout: 10000,
  headers: {'X-Version': '2', 'X-New': 'value'}
}

// Result:
// - requestTimeout: 10000 (overridden)
// - headers: {'X-Version': '2', 'X-New': 'value'} (merged)
await req('https://api.example.com/data', options1, options2)
```

**Combining retry strategies:**
```typescript
const networkRetry = {
  retry: {codes: ['ECONNRESET'], limit: 3, timeout: 500}
}

const serverRetry = {
  retry: {statuses: [500, 503], limit: 2, timeout: 2000}
}

// Both retry strategies will be active
await req('https://api.example.com/data', networkRetry, serverRetry)
```

**Accumulating hooks:**
```typescript
const loggingHooks = {
  hooks: {
    beforeRequest: ({request}) => console.log('Request:', request.url)
  }
}

const authHooks = {
  hooks: {
    beforeRequest: ({request}) => request.headers.set('Auth', 'token')
  }
}

// Both hooks execute in order
await req('https://api.example.com/data', loggingHooks, authHooks)
```

**Practical composition pattern:**
```typescript
// Define reusable option sets
const apiDefaults = {
  baseUrl: 'https://api.example.com',
  connectionTimeout: 30000,
  retry: {codes: ['ECONNRESET'], limit: 3}
}

const withAuth = {
  headers: {'Authorization': `Bearer ${token}`}
}

const withRetry = {
  retry: {statuses: [500, 502, 503], limit: 5, timeout: [1000, 2000, 4000]}
}

const withLogging = {
  hooks: {
    beforeRequest: ({request}) => logger.info('Request', request.url),
    afterResponse: ({response}) => logger.info('Response', response.status)
  }
}

// Compose as needed
await req('/users', apiDefaults, withAuth)
await req('/critical-data', apiDefaults, withAuth, withRetry, withLogging)
```

**Using makeReq for composition:**
```typescript
// Create base client
const apiClient = makeReq({
  baseUrl: 'https://api.example.com',
  headers: {'User-Agent': 'MyApp/1.0'},
  retry: {codes: ['ECONNRESET']}
})

// Add authentication per request
await apiClient('/public-data')
await apiClient('/private-data', {
  headers: {'Authorization': 'Bearer token'}
})

// Create specialized client with additional options
const authedClient = makeReq({
  baseUrl: 'https://api.example.com',
  headers: {
    'User-Agent': 'MyApp/1.0',
    'Authorization': 'Bearer token'
  }
})

await authedClient('/user/profile')
```

## Examples

### Complete real-world example

```typescript
import {req, makeReq, stop} from '@applitools/req'

// Create API client with defaults
const apiClient = makeReq({
  baseUrl: 'https://api.example.com/v1',
  connectionTimeout: 60000,
  requestTimeout: 10000,
  headers: {
    'User-Agent': 'MyApp/2.0',
    'Accept': 'application/json'
  },
  retry: [
    // Quick retry for network errors
    {
      codes: ['ECONNRESET', 'ETIMEDOUT'],
      limit: 3,
      timeout: 500
    },
    // Slower retry for server errors
    {
      statuses: [500, 502, 503],
      limit: 5,
      timeout: [1000, 2000, 4000, 8000]
    }
  ],
  hooks: {
    beforeRequest: ({request}) => {
      console.log(`→ ${request.method} ${request.url}`)
    },
    afterResponse: ({response}) => {
      console.log(`← ${response.status} ${response.statusText}`)
    },
    afterError: ({error}) => {
      console.error('Request failed:', error.message)
    }
  }
})

// Authenticated requests
const authedClient = makeReq({
  baseUrl: 'https://api.example.com/v1',
  headers: {'Authorization': `Bearer ${getToken()}`},
  retry: {
    statuses: [401],
    limit: 1
  },
  hooks: {
    beforeRetry: async ({request, response, stop}) => {
      if (response?.status === 401) {
        try {
          const newToken = await refreshToken()
          request.headers.set('Authorization', `Bearer ${newToken}`)
          return request
        } catch {
          return stop
        }
      }
    }
  }
})

// Usage
const users = await apiClient('/users', {
  query: {page: 1, limit: 20}
})

const profile = await authedClient('/user/profile')

const result = await authedClient('/user/update', {
  method: 'POST',
  body: {name: 'John Doe', email: 'john@example.com'}
})
```

## License

MIT